[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:43:50 dist-test-slave-mn38 krb5kdc[9074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:43:50 dist-test-slave-mn38 krb5kdc[9074](info): set up 2 sockets
Sep 28 10:43:50 dist-test-slave-mn38 krb5kdc[9074](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 10:43:51.490801 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:43:51 dist-test-slave-mn38 krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:43:51.673978 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:39945
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:43:52.020797 9087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:43:52.021171 9087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:43:52.021296 9087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:43:52.029173 9087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:43:52.029340 9087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:43:52.029415 9087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:43:52.029525 9087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:43:52.029601 9087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:43:52.040220 9087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:39945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:43:52.041697 9087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:43:52.043124 9087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:43:52.055553 9092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:52.055585 9093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:52.055621 9095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:52.056187 9087 server_base.cc:1034] running on GCE node
I20260928 10:43:52.057143 9087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:43:52.058746 9087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:43:52.059966 9087 hybrid_clock.cc:648] HybridClock initialized: now 1790592232059916 us; error 45 us; skew 500 ppm
I20260928 10:43:52.060384 9087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:43:52 dist-test-slave-mn38 krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592232, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:43:52.063529 9087 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:43:52.065307 9087 webserver.cc:460] Webserver started at http://127.8.218.254:43689/ using document root <none> and password file <none>
I20260928 10:43:52.066373 9087 fs_manager.cc:359] Metadata directory not provided
I20260928 10:43:52.066514 9087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:43:52.066805 9087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:43:52.070232 9087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "9765c1243f544241b93614ae0978ee6b"
format_stamp: "Formatted at 2026-09-28 10:43:52 on dist-test-slave-mn38"
server_key: "af7da51a64091601fbfd060d5de78d17"
server_key_iv: "2a14d340d3c214c7120e1e5e4aebf86b"
server_key_version: "encryptionkey@0"
I20260928 10:43:52.071079 9087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "9765c1243f544241b93614ae0978ee6b"
format_stamp: "Formatted at 2026-09-28 10:43:52 on dist-test-slave-mn38"
server_key: "af7da51a64091601fbfd060d5de78d17"
server_key_iv: "2a14d340d3c214c7120e1e5e4aebf86b"
server_key_version: "encryptionkey@0"
I20260928 10:43:52.077347 9087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:43:52.082338 9102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:52.084089 9087 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:43:52.084307 9087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "9765c1243f544241b93614ae0978ee6b"
format_stamp: "Formatted at 2026-09-28 10:43:52 on dist-test-slave-mn38"
server_key: "af7da51a64091601fbfd060d5de78d17"
server_key_iv: "2a14d340d3c214c7120e1e5e4aebf86b"
server_key_version: "encryptionkey@0"
I20260928 10:43:52.084532 9087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:43:52.119415 9087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:43:52.124447 9087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:43:52.124758 9087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:43:52.142869 9087 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:39945
I20260928 10:43:52.142901 9153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:39945 every 8 connection(s)
I20260928 10:43:52.144294 9087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:43:52.148594 9154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:43:52.152895 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 9087
I20260928 10:43:52.153221 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:43:52.155265 9067 external_mini_cluster.cc:1428] Setting key 85578f304e233c2bd1d72c2777cda73d
I20260928 10:43:52.159855 9154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b: Bootstrap starting.
I20260928 10:43:52.163385 9154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b: Neither blocks nor log segments found. Creating new log.
I20260928 10:43:52.164453 9154 log.cc:824] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:43:52.167824 9154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b: No bootstrap required, opened a new log
I20260928 10:43:52.177990 9154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9765c1243f544241b93614ae0978ee6b" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39945 } }
I20260928 10:43:52.178442 9154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:43:52.178557 9154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9765c1243f544241b93614ae0978ee6b, State: Initialized, Role: FOLLOWER
I20260928 10:43:52.179379 9154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9765c1243f544241b93614ae0978ee6b" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39945 } }
I20260928 10:43:52.179746 9154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:43:52.179900 9154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:43:52.180097 9154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:43:52.182855 9154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9765c1243f544241b93614ae0978ee6b" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39945 } }
I20260928 10:43:52.183598 9154 leader_election.cc:302] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9765c1243f544241b93614ae0978ee6b; no voters:
I20260928 10:43:52.184381 9154 leader_election.cc:288] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:43:52.184615 9159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:43:52.186293 9159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [term 1 LEADER]: Becoming Leader. State: Replica: 9765c1243f544241b93614ae0978ee6b, State: Running, Role: LEADER
I20260928 10:43:52.187129 9159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9765c1243f544241b93614ae0978ee6b" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39945 } }
I20260928 10:43:52.189354 9154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:43:52.191397 9160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9765c1243f544241b93614ae0978ee6b. Latest consensus state: current_term: 1 leader_uuid: "9765c1243f544241b93614ae0978ee6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9765c1243f544241b93614ae0978ee6b" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39945 } } }
I20260928 10:43:52.192317 9160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [sys.catalog]: This master's current role is: LEADER
Sep 28 10:43:52 dist-test-slave-mn38 krb5kdc[9074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:43:52.196455 9161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9765c1243f544241b93614ae0978ee6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9765c1243f544241b93614ae0978ee6b" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39945 } } }
I20260928 10:43:52.196904 9161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b [sys.catalog]: This master's current role is: LEADER
I20260928 10:43:52.202657 9164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:43:52.210170 9164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:43:52.210567 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:52.159491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.8.218.254:39945)
0928 10:43:52.160517 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:52.160535 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:52.162008 (+ 1473us) server_negotiation.cc:407] Connection header received
0928 10:43:52.173126 (+ 11118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:52.173184 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:52.173997 (+ 813us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:52.174627 (+ 630us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:52.178650 (+ 4023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:52.180715 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:52.183027 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:52.184347 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:52.185357 (+ 1010us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:52.195404 (+ 10047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:52.195419 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:52.195450 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:52.195895 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:52.201512 (+ 5617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:52.203343 (+ 1831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:52.203360 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:52.203373 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:52.203724 (+ 351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:52.204364 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:52.204368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:52.204373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:52.205007 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:52.205306 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:52.208355 (+ 3049us) server_negotiation.cc:299] Negotiation successful
0928 10:43:52.208924 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":478,"threads_started":1}
I20260928 10:43:52.224340 9164 catalog_manager.cc:1347] Generated new cluster ID: bfa1fff58ee6437799ab8b05c5d8458d
I20260928 10:43:52.224481 9164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:43:52.253104 9164 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:43:52.254627 9164 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:43:52.273744 9164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9765c1243f544241b93614ae0978ee6b: Generated new TSK 0
I20260928 10:43:52.274806 9164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:43:52.336009 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:39945
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:43:52.698421 9180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:43:52.698760 9180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:43:52.698904 9180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:43:52.707209 9180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:43:52.707373 9180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:43:52.707537 9180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:43:52.718076 9180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:39945
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:43:52.719470 9180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:43:52.720932 9180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:43:52.734954 9186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:52.734931 9188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:52.735206 9185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:52.735759 9180 server_base.cc:1034] running on GCE node
I20260928 10:43:52.736431 9180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:43:52.737429 9180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:43:52.738710 9180 hybrid_clock.cc:648] HybridClock initialized: now 1790592232738649 us; error 49 us; skew 500 ppm
I20260928 10:43:52.739034 9180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:43:52 dist-test-slave-mn38 krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592232, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:43:52.742206 9180 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:43:52.743692 9180 webserver.cc:460] Webserver started at http://127.8.218.193:40167/ using document root <none> and password file <none>
I20260928 10:43:52.744390 9180 fs_manager.cc:359] Metadata directory not provided
I20260928 10:43:52.744532 9180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:43:52.744823 9180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:43:52.747859 9180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "3c95fd79a9f74d89abae6d7d8704a7fd"
format_stamp: "Formatted at 2026-09-28 10:43:52 on dist-test-slave-mn38"
server_key: "62271367eaf1da1818ad5bf4c362ba48"
server_key_iv: "10721fbfe79e869400512ae3a414e521"
server_key_version: "encryptionkey@0"
I20260928 10:43:52.748646 9180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "3c95fd79a9f74d89abae6d7d8704a7fd"
format_stamp: "Formatted at 2026-09-28 10:43:52 on dist-test-slave-mn38"
server_key: "62271367eaf1da1818ad5bf4c362ba48"
server_key_iv: "10721fbfe79e869400512ae3a414e521"
server_key_version: "encryptionkey@0"
I20260928 10:43:52.754521 9180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:43:52.758347 9195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:52.759914 9180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:43:52.760131 9180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "3c95fd79a9f74d89abae6d7d8704a7fd"
format_stamp: "Formatted at 2026-09-28 10:43:52 on dist-test-slave-mn38"
server_key: "62271367eaf1da1818ad5bf4c362ba48"
server_key_iv: "10721fbfe79e869400512ae3a414e521"
server_key_version: "encryptionkey@0"
I20260928 10:43:52.760346 9180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:43:52.793531 9180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:43:52.798664 9180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:43:52.798975 9180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:43:52.800062 9180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:43:52.801684 9180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:43:52.801771 9180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:52.801905 9180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:43:52.801971 9180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:52.833777 9180 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:42251
I20260928 10:43:52.833817 9307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:42251 every 8 connection(s)
I20260928 10:43:52.835173 9180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:43:52.836630 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 9180
I20260928 10:43:52.836862 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:43:52.837656 9067 external_mini_cluster.cc:1428] Setting key 480d394dc0dbf032328771dee9489062
Sep 28 10:43:52 dist-test-slave-mn38 krb5kdc[9074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592232, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
I20260928 10:43:52.857754 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:52.838438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:53013 (local address 127.8.218.254:39945)
0928 10:43:52.838911 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:52.838916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:52.840201 (+ 1285us) server_negotiation.cc:407] Connection header received
0928 10:43:52.842137 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:52.842142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:52.842225 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:52.842331 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:52.844028 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:52.845119 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:52.846898 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:52.847526 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:52.847657 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:52.851553 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:52.851560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:52.851565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:52.851610 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:52.854383 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:52.855242 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:52.855247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:52.855251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:52.855397 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:52.855883 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:52.855888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:52.855891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:52.856260 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:52.856403 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:52.857181 (+ 778us) server_negotiation.cc:299] Negotiation successful
0928 10:43:52.857399 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260928 10:43:52.858844 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:52.839155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:39945 (local address 127.8.218.193:53013)
0928 10:43:52.840028 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:52.840084 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:52.841866 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:52.842674 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:52.842686 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:52.843233 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:43:52.843835 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:52.843856 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:52.845303 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:52.845308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:52.846747 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:52.846763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:52.847766 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:52.847772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:52.847998 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:52.848746 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:43:52.848771 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:43:52.851284 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:43:52.854599 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:52.854609 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:52.854630 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:52.855102 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:52.855575 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:52.855579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:52.855581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:52.855770 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:52.856392 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:43:52.856398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:43:52.856801 (+ 403us) client_negotiation.cc:769] Sending connection context
0928 10:43:52.857033 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 10:43:52.857464 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":352,"threads_started":1}
I20260928 10:43:52.860571 9308 heartbeater.cc:347] Connected to a master server at 127.8.218.254:39945
I20260928 10:43:52.860864 9308 heartbeater.cc:464] Registering TS with master...
I20260928 10:43:52.861639 9308 heartbeater.cc:511] Master 127.8.218.254:39945 requested a full tablet report, sending...
I20260928 10:43:52.863849 9119 ts_manager.cc:194] Registered new tserver with Master: 3c95fd79a9f74d89abae6d7d8704a7fd (127.8.218.193:42251)
I20260928 10:43:52.865415 9119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:53013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:43:52.877763 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:39945
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:43:53.226332 9314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:43:53.226686 9314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:43:53.226822 9314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:43:53.234666 9314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:43:53.234838 9314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:43:53.235002 9314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:43:53.247040 9314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:39945
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:43:53.248862 9314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:43:53.250468 9314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:43:53.265702 9319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:53.265579 9320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:53.265558 9322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:53.266345 9314 server_base.cc:1034] running on GCE node
I20260928 10:43:53.267035 9314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:43:53.268168 9314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:43:53.269405 9314 hybrid_clock.cc:648] HybridClock initialized: now 1790592233269332 us; error 64 us; skew 500 ppm
I20260928 10:43:53.269891 9314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:43:53 dist-test-slave-mn38 krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:43:53.273404 9314 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:43:53.275506 9314 webserver.cc:460] Webserver started at http://127.8.218.194:35651/ using document root <none> and password file <none>
I20260928 10:43:53.276335 9314 fs_manager.cc:359] Metadata directory not provided
I20260928 10:43:53.276479 9314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:43:53.276790 9314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:43:53.280237 9314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "020e4f5fa9f74b98883d59c81feea8b3"
format_stamp: "Formatted at 2026-09-28 10:43:53 on dist-test-slave-mn38"
server_key: "4879f54c27a30478e618ce15f8ac1370"
server_key_iv: "9a54e5c8319e530f3f42cd08ff922756"
server_key_version: "encryptionkey@0"
I20260928 10:43:53.281096 9314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "020e4f5fa9f74b98883d59c81feea8b3"
format_stamp: "Formatted at 2026-09-28 10:43:53 on dist-test-slave-mn38"
server_key: "4879f54c27a30478e618ce15f8ac1370"
server_key_iv: "9a54e5c8319e530f3f42cd08ff922756"
server_key_version: "encryptionkey@0"
I20260928 10:43:53.287297 9314 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 10:43:53.292048 9329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:53.294310 9314 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:43:53.294552 9314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "020e4f5fa9f74b98883d59c81feea8b3"
format_stamp: "Formatted at 2026-09-28 10:43:53 on dist-test-slave-mn38"
server_key: "4879f54c27a30478e618ce15f8ac1370"
server_key_iv: "9a54e5c8319e530f3f42cd08ff922756"
server_key_version: "encryptionkey@0"
I20260928 10:43:53.294816 9314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:43:53.318295 9314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:43:53.324119 9314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:43:53.324491 9314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:43:53.325712 9314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:43:53.327407 9314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:43:53.327519 9314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:53.327651 9314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:43:53.327705 9314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:53.360555 9314 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:38537
I20260928 10:43:53.360641 9441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:38537 every 8 connection(s)
I20260928 10:43:53.362124 9314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:43:53.366290 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 9314
I20260928 10:43:53.366503 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:43:53.367033 9067 external_mini_cluster.cc:1428] Setting key 6253df660d892e52cc32e43fd286395a
Sep 28 10:43:53 dist-test-slave-mn38 krb5kdc[9074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:43:53.386111 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:53.364802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:37525 (local address 127.8.218.254:39945)
0928 10:43:53.365213 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:53.365217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:53.366574 (+ 1357us) server_negotiation.cc:407] Connection header received
0928 10:43:53.368938 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:53.368942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:53.369016 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:53.369135 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:53.370991 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.372023 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.373954 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.374390 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.374503 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:53.379164 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:53.379171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:53.379175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:53.379224 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:53.382172 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:53.383409 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:53.383414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:53.383416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:53.383602 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:53.384108 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:53.384112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:53.384115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:53.384475 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:53.384647 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:53.385581 (+ 934us) server_negotiation.cc:299] Negotiation successful
0928 10:43:53.385815 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":219,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
I20260928 10:43:53.387619 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:53.365454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:39945 (local address 127.8.218.194:37525)
0928 10:43:53.366368 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:53.366428 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:53.368610 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:53.369478 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:53.369492 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:53.370124 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:43:53.370769 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:53.370790 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.372244 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.372249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:53.373795 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:53.373814 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.374635 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.374642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:53.374947 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:53.375706 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:43:53.375737 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:43:53.378789 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:43:53.382395 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:53.382408 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:53.382431 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:53.383234 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:53.383726 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:53.383732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:53.383735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:53.383949 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:53.384648 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:43:53.384656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:43:53.385061 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 10:43:53.385325 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 10:43:53.385954 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":285,"threads_started":1}
I20260928 10:43:53.389544 9442 heartbeater.cc:347] Connected to a master server at 127.8.218.254:39945
I20260928 10:43:53.389956 9442 heartbeater.cc:464] Registering TS with master...
I20260928 10:43:53.390946 9442 heartbeater.cc:511] Master 127.8.218.254:39945 requested a full tablet report, sending...
I20260928 10:43:53.393474 9119 ts_manager.cc:194] Registered new tserver with Master: 020e4f5fa9f74b98883d59c81feea8b3 (127.8.218.194:38537)
I20260928 10:43:53.394614 9119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:37525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:43:53.411329 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:39945
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:43:53.758796 9448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:43:53.759112 9448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:43:53.759209 9448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:43:53.766876 9448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:43:53.767010 9448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:43:53.767136 9448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:43:53.777772 9448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:39945
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:43:53.779306 9448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:43:53.781183 9448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:43:53.797986 9454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:53.798254 9453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:53.798273 9456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:53.800424 9448 server_base.cc:1034] running on GCE node
I20260928 10:43:53.801230 9448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:43:53.802419 9448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:43:53.803653 9448 hybrid_clock.cc:648] HybridClock initialized: now 1790592233803596 us; error 55 us; skew 500 ppm
I20260928 10:43:53.803992 9448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:43:53 dist-test-slave-mn38 krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:43:53.807449 9448 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:43:53.809211 9448 webserver.cc:460] Webserver started at http://127.8.218.195:44625/ using document root <none> and password file <none>
I20260928 10:43:53.810078 9448 fs_manager.cc:359] Metadata directory not provided
I20260928 10:43:53.810222 9448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:43:53.810552 9448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:43:53.813899 9448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "208903d2fa5947aaa62f548e777e6d69"
format_stamp: "Formatted at 2026-09-28 10:43:53 on dist-test-slave-mn38"
server_key: "65542a6b51524323aec0f60ddd6c66c7"
server_key_iv: "334127f197b8fe3989a816f4ffc357ff"
server_key_version: "encryptionkey@0"
I20260928 10:43:53.814776 9448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "208903d2fa5947aaa62f548e777e6d69"
format_stamp: "Formatted at 2026-09-28 10:43:53 on dist-test-slave-mn38"
server_key: "65542a6b51524323aec0f60ddd6c66c7"
server_key_iv: "334127f197b8fe3989a816f4ffc357ff"
server_key_version: "encryptionkey@0"
I20260928 10:43:53.820786 9448 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:43:53.824896 9463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:53.826715 9448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:43:53.826938 9448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "208903d2fa5947aaa62f548e777e6d69"
format_stamp: "Formatted at 2026-09-28 10:43:53 on dist-test-slave-mn38"
server_key: "65542a6b51524323aec0f60ddd6c66c7"
server_key_iv: "334127f197b8fe3989a816f4ffc357ff"
server_key_version: "encryptionkey@0"
I20260928 10:43:53.827173 9448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:43:53.848748 9448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:43:53.853116 9448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:43:53.853387 9448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:43:53.854418 9448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:43:53.855822 9448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:43:53.855901 9448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:53.856019 9448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:43:53.856070 9448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:53.869114 9308 heartbeater.cc:503] Master 127.8.218.254:39945 was elected leader, sending a full tablet report...
I20260928 10:43:53.888432 9448 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:40981
I20260928 10:43:53.888520 9575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:40981 every 8 connection(s)
I20260928 10:43:53.890084 9448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:43:53.899653 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 9448
I20260928 10:43:53.899832 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:43:53.900338 9067 external_mini_cluster.cc:1428] Setting key 4f7e00417b78690984eadc27f7464ced
Sep 28 10:43:53 dist-test-slave-mn38 krb5kdc[9074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:43:53.912771 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:53.893011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:46343 (local address 127.8.218.254:39945)
0928 10:43:53.893507 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:53.893513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:53.894669 (+ 1156us) server_negotiation.cc:407] Connection header received
0928 10:43:53.896546 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:53.896552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:53.896706 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:53.896805 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:53.898586 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.899425 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.901403 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.902012 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.902128 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:53.906311 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:53.906316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:53.906320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:53.906430 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:53.909145 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:53.910238 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:53.910243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:53.910246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:53.910412 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:53.910892 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:53.910896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:53.910899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:53.911184 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:53.911311 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:53.912250 (+ 939us) server_negotiation.cc:299] Negotiation successful
0928 10:43:53.912466 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260928 10:43:53.913924 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:53.893722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:39945 (local address 127.8.218.195:46343)
0928 10:43:53.894489 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:53.894535 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:53.896234 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:53.897108 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:53.897118 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:53.897746 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:43:53.898388 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:53.898409 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.899636 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.899641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:53.901254 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:53.901272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.902237 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.902242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:53.902483 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:53.903294 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:43:53.903333 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:43:53.906074 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:43:53.909347 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:53.909357 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:53.909375 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:53.910069 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:53.910563 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:53.910567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:53.910570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:53.910778 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:53.911343 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:43:53.911351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:43:53.911777 (+ 426us) client_negotiation.cc:769] Sending connection context
0928 10:43:53.912063 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 10:43:53.912597 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":306,"threads_started":1}
I20260928 10:43:53.915441 9576 heartbeater.cc:347] Connected to a master server at 127.8.218.254:39945
I20260928 10:43:53.915768 9576 heartbeater.cc:464] Registering TS with master...
I20260928 10:43:53.916591 9576 heartbeater.cc:511] Master 127.8.218.254:39945 requested a full tablet report, sending...
I20260928 10:43:53.918270 9119 ts_manager.cc:194] Registered new tserver with Master: 208903d2fa5947aaa62f548e777e6d69 (127.8.218.195:40981)
I20260928 10:43:53.919114 9119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:46343
I20260928 10:43:53.929796 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:43:53 dist-test-slave-mn38 krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:43:53 dist-test-slave-mn38 krb5kdc[9074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:43:53.975518 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:53.958210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.8.218.193:42251)
0928 10:43:53.958900 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:53.958908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:53.958936 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:43:53.958998 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:53.959004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:53.959229 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:53.959363 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:53.960912 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.962113 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.963506 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:53.964155 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:53.964270 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:53.968636 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:53.968656 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:53.968675 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:53.968726 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:53.971921 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:53.972828 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:53.972837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:53.972840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:53.973004 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:53.973545 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:53.973552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:53.973555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:53.974059 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:53.974376 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:53.974783 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 10:43:53.975082 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":375,"threads_started":1}
W20260928 10:43:53.976982 9223 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37846
Sep 28 10:43:53 dist-test-slave-mn38 krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:43:54 dist-test-slave-mn38 krb5kdc[9074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:43:54.021243 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:54.007183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.8.218.193:42251)
0928 10:43:54.007571 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:54.007578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:54.007725 (+ 147us) server_negotiation.cc:407] Connection header received
0928 10:43:54.007879 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:54.007883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:54.007985 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:54.008087 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:54.009566 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:54.010278 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:54.011656 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:54.012140 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:54.012258 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:54.016246 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:54.016252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:54.016255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:54.016299 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:54.018983 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:54.019699 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:54.019703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:54.019705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:54.019833 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:54.020149 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:54.020152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:54.020154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:54.020400 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:54.020575 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:54.020815 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 10:43:54.020980 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260928 10:43:54.022975 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 9180
I20260928 10:43:54.038300 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:54.024014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.8.218.193:42251)
0928 10:43:54.024234 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:54.024239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:54.024449 (+ 210us) server_negotiation.cc:407] Connection header received
0928 10:43:54.024583 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:54.024587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:54.024657 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:54.024738 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:54.026120 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:54.026937 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:54.028792 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:54.029379 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:54.029796 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:54.032381 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:54.032388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:54.032393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:54.032449 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:54.035218 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:54.036156 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:54.036161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:54.036164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:54.036341 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:54.036788 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:54.036791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:54.036793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:54.037118 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:54.037346 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:54.037731 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 10:43:54.037951 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:43:54.039142 9303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:43:54.124066 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 9180
I20260928 10:43:54.139627 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 9314
Sep 28 10:43:54 dist-test-slave-mn38 krb5kdc[9074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:43:54.157799 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:54.140638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.8.218.194:38537)
0928 10:43:54.141208 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:54.141215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:54.141242 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:43:54.141300 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:54.141307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:54.141574 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:54.141709 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:54.142992 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:54.144627 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:54.146056 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:54.146976 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:54.147134 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:54.151659 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:54.151677 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:54.151694 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:54.151748 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:54.154698 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:54.155455 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:54.155464 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:54.155466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:54.155613 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:54.156073 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:54.156077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:54.156079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:54.156465 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:54.156697 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:54.157114 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 10:43:54.157341 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":315,"threads_started":1}
I20260928 10:43:54.159219 9437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:43:54.243706 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 9314
I20260928 10:43:54.262162 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 9448
Sep 28 10:43:54 dist-test-slave-mn38 krb5kdc[9074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:43:54.280021 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:54.263174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.8.218.195:40981)
0928 10:43:54.263840 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:54.263847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:54.263876 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:43:54.263966 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:54.263973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:54.264193 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:54.264364 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:54.265653 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:54.266715 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:54.268054 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:54.268651 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:54.268772 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:54.273200 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:54.273217 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:54.273234 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:54.273283 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:54.276098 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:54.277193 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:54.277205 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:54.277208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:54.277408 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:54.277923 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:54.277927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:54.277930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:54.278306 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:54.278616 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:54.279094 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 10:43:54.279719 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":370,"threads_started":1}
I20260928 10:43:54.281818 9571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:43:54.365334 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 9448
I20260928 10:43:54.379611 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 9087
I20260928 10:43:54.380936 9149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:43:54.487967 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 9087
2026-09-28T10:43:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4039 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:43:54 dist-test-slave-mn38 krb5kdc[9598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:43:54 dist-test-slave-mn38 krb5kdc[9598](info): set up 2 sockets
Sep 28 10:43:54 dist-test-slave-mn38 krb5kdc[9598](info): commencing operation
krb5kdc: starting...
W20260928 10:43:55.511193 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:43:55 dist-test-slave-mn38 krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:43:55.695256 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45215
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:43:56.038362 9611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:43:56.038710 9611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:43:56.038828 9611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:43:56.046465 9611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:43:56.046610 9611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:43:56.046657 9611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:43:56.046720 9611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:43:56.046785 9611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:43:56.057740 9611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:43:56.059222 9611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:43:56.060616 9611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:43:56.073295 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:56.073261 9619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:56.073316 9617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:56.073860 9611 server_base.cc:1034] running on GCE node
I20260928 10:43:56.074880 9611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:43:56.076363 9611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:43:56.077555 9611 hybrid_clock.cc:648] HybridClock initialized: now 1790592236077487 us; error 53 us; skew 500 ppm
I20260928 10:43:56.077945 9611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:43:56 dist-test-slave-mn38 krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592236, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:43:56.085137 9611 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:43:56.086902 9611 webserver.cc:460] Webserver started at http://127.8.218.254:35989/ using document root <none> and password file <none>
I20260928 10:43:56.087721 9611 fs_manager.cc:359] Metadata directory not provided
I20260928 10:43:56.087845 9611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:43:56.088192 9611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:43:56.091534 9611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "269b19a573b241ae8368502fd23200a5"
format_stamp: "Formatted at 2026-09-28 10:43:56 on dist-test-slave-mn38"
server_key: "2e152c55a53fe70f015c815706d8661a"
server_key_iv: "afdcd6dd2283cb704702053084f0af9a"
server_key_version: "encryptionkey@0"
I20260928 10:43:56.092417 9611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "269b19a573b241ae8368502fd23200a5"
format_stamp: "Formatted at 2026-09-28 10:43:56 on dist-test-slave-mn38"
server_key: "2e152c55a53fe70f015c815706d8661a"
server_key_iv: "afdcd6dd2283cb704702053084f0af9a"
server_key_version: "encryptionkey@0"
I20260928 10:43:56.098547 9611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:43:56.102360 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:56.103859 9611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:43:56.104053 9611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "269b19a573b241ae8368502fd23200a5"
format_stamp: "Formatted at 2026-09-28 10:43:56 on dist-test-slave-mn38"
server_key: "2e152c55a53fe70f015c815706d8661a"
server_key_iv: "afdcd6dd2283cb704702053084f0af9a"
server_key_version: "encryptionkey@0"
I20260928 10:43:56.104246 9611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:43:56.117843 9611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:43:56.122645 9611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:43:56.122994 9611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:43:56.142673 9611 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:45215
I20260928 10:43:56.142679 9677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:45215 every 8 connection(s)
I20260928 10:43:56.144212 9611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:43:56.148653 9678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:43:56.153529 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 9611
I20260928 10:43:56.153774 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:43:56.154587 9067 external_mini_cluster.cc:1428] Setting key 043f067f8f15cd252b76ab7d2cf24c30
I20260928 10:43:56.162088 9678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5: Bootstrap starting.
I20260928 10:43:56.166289 9678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5: Neither blocks nor log segments found. Creating new log.
I20260928 10:43:56.167635 9678 log.cc:824] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:43:56 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:43:56.171737 9678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5: No bootstrap required, opened a new log
I20260928 10:43:56.182372 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:56.156954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.8.218.254:45215)
0928 10:43:56.157879 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:56.157893 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:56.157950 (+ 57us) server_negotiation.cc:407] Connection header received
0928 10:43:56.160173 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:56.160209 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:56.160774 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:56.161217 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:56.163169 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:56.164686 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:56.166580 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:56.167463 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:56.167637 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:56.172517 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:56.172527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:56.172546 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:56.172597 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:56.176198 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:56.177465 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:56.177483 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:56.177502 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:56.177776 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:56.178387 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:56.178392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:56.178394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:56.179400 (+ 1006us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:56.179676 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:56.180520 (+ 844us) server_negotiation.cc:299] Negotiation successful
0928 10:43:56.181064 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":435,"threads_started":1}
I20260928 10:43:56.182240 9678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269b19a573b241ae8368502fd23200a5" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45215 } }
I20260928 10:43:56.182794 9678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:43:56.182907 9678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b19a573b241ae8368502fd23200a5, State: Initialized, Role: FOLLOWER
I20260928 10:43:56.183715 9678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269b19a573b241ae8368502fd23200a5" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45215 } }
I20260928 10:43:56.184036 9678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:43:56.184180 9678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:43:56.184357 9678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:43:56.187317 9678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269b19a573b241ae8368502fd23200a5" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45215 } }
I20260928 10:43:56.188007 9678 leader_election.cc:302] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 269b19a573b241ae8368502fd23200a5; no voters:
I20260928 10:43:56.188889 9678 leader_election.cc:288] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:43:56.188969 9683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:43:56.190349 9683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [term 1 LEADER]: Becoming Leader. State: Replica: 269b19a573b241ae8368502fd23200a5, State: Running, Role: LEADER
I20260928 10:43:56.191054 9683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269b19a573b241ae8368502fd23200a5" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45215 } }
I20260928 10:43:56.191704 9678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:43:56.194039 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "269b19a573b241ae8368502fd23200a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269b19a573b241ae8368502fd23200a5" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45215 } } }
I20260928 10:43:56.194411 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [sys.catalog]: This master's current role is: LEADER
I20260928 10:43:56.194811 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 269b19a573b241ae8368502fd23200a5. Latest consensus state: current_term: 1 leader_uuid: "269b19a573b241ae8368502fd23200a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269b19a573b241ae8368502fd23200a5" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45215 } } }
I20260928 10:43:56.195240 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5 [sys.catalog]: This master's current role is: LEADER
I20260928 10:43:56.198308 9688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:43:56.205101 9688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:43:56.220016 9688 catalog_manager.cc:1347] Generated new cluster ID: c9843cf45f4d4e89ac47ad822fb059db
I20260928 10:43:56.220146 9688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:43:56.245157 9688 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:43:56.246596 9688 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:43:56.254165 9688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 269b19a573b241ae8368502fd23200a5: Generated new TSK 0
I20260928 10:43:56.255133 9688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:43:56.310217 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45215
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:43:56.659747 9704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:43:56.660048 9704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:43:56.660146 9704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:43:56.667630 9704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:43:56.667758 9704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:43:56.667876 9704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:43:56.678427 9704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:43:56.679754 9704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:43:56.681200 9704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:43:56.695096 9709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:56.695271 9712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:56.695102 9710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:56.696429 9704 server_base.cc:1034] running on GCE node
I20260928 10:43:56.697111 9704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:43:56.698274 9704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:43:56.699486 9704 hybrid_clock.cc:648] HybridClock initialized: now 1790592236699413 us; error 57 us; skew 500 ppm
I20260928 10:43:56.699929 9704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:43:56 dist-test-slave-mn38 krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592236, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:43:56.703361 9704 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:43:56.704968 9704 webserver.cc:460] Webserver started at http://127.8.218.193:33455/ using document root <none> and password file <none>
I20260928 10:43:56.705806 9704 fs_manager.cc:359] Metadata directory not provided
I20260928 10:43:56.705940 9704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:43:56.706254 9704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:43:56.709503 9704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "9665104697c3493eb909a1108407fc49"
format_stamp: "Formatted at 2026-09-28 10:43:56 on dist-test-slave-mn38"
server_key: "6352a6f4c14d66dc036d3c5db250ebb2"
server_key_iv: "e8c84286ce06c41081d6acc3ea066d27"
server_key_version: "encryptionkey@0"
I20260928 10:43:56.710374 9704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "9665104697c3493eb909a1108407fc49"
format_stamp: "Formatted at 2026-09-28 10:43:56 on dist-test-slave-mn38"
server_key: "6352a6f4c14d66dc036d3c5db250ebb2"
server_key_iv: "e8c84286ce06c41081d6acc3ea066d27"
server_key_version: "encryptionkey@0"
I20260928 10:43:56.715982 9704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:43:56.720062 9719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:56.721758 9704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:43:56.721944 9704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "9665104697c3493eb909a1108407fc49"
format_stamp: "Formatted at 2026-09-28 10:43:56 on dist-test-slave-mn38"
server_key: "6352a6f4c14d66dc036d3c5db250ebb2"
server_key_iv: "e8c84286ce06c41081d6acc3ea066d27"
server_key_version: "encryptionkey@0"
I20260928 10:43:56.722144 9704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:43:56.748831 9704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:43:56.753858 9704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:43:56.754142 9704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:43:56.755147 9704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:43:56.756625 9704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:43:56.756713 9704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:56.756831 9704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:43:56.756881 9704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:56.790473 9704 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:44339
I20260928 10:43:56.790521 9831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:44339 every 8 connection(s)
I20260928 10:43:56.792416 9704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:43:56.799672 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 9704
I20260928 10:43:56.800081 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:43:56.800900 9067 external_mini_cluster.cc:1428] Setting key 49788cdeeb674cf629471677987ac198
Sep 28 10:43:56 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592236, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:43:56.822782 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:56.797096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:35271 (local address 127.8.218.254:45215)
0928 10:43:56.797813 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:56.797820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:56.801532 (+ 3712us) server_negotiation.cc:407] Connection header received
0928 10:43:56.802445 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:56.802451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:56.802547 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:56.802655 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:56.805384 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:56.806562 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:56.809375 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:56.810083 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:56.810248 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:56.815488 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:56.815497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:56.815502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:56.815550 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:56.819390 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:56.820338 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:56.820345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:56.820349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:56.820498 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:56.821103 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:56.821109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:56.821112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:56.821396 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:56.821572 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:56.822340 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 10:43:56.822521 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":274,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
I20260928 10:43:56.824043 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:56.798176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45215 (local address 127.8.218.193:35271)
0928 10:43:56.799561 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:56.799624 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:56.802117 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:56.803024 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:56.803041 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:56.803879 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:43:56.805102 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:56.805140 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:56.806800 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:56.806806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:56.809041 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:56.809069 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:56.810653 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:56.810661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:56.811046 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:56.811870 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:43:56.811905 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:43:56.815156 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:43:56.819621 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:56.819630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:56.819648 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:56.820181 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:56.820670 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:56.820674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:56.820676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:56.820983 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:56.821566 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:43:56.821573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:43:56.821999 (+ 426us) client_negotiation.cc:769] Sending connection context
0928 10:43:56.822255 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 10:43:56.822700 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":748,"threads_started":1}
I20260928 10:43:56.825846 9832 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45215
I20260928 10:43:56.826184 9832 heartbeater.cc:464] Registering TS with master...
I20260928 10:43:56.827033 9832 heartbeater.cc:511] Master 127.8.218.254:45215 requested a full tablet report, sending...
I20260928 10:43:56.829530 9643 ts_manager.cc:194] Registered new tserver with Master: 9665104697c3493eb909a1108407fc49 (127.8.218.193:44339)
I20260928 10:43:56.831614 9643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:35271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:43:56.847400 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45215
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:43:57.189940 9838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:43:57.190289 9838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:43:57.190433 9838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:43:57.198318 9838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:43:57.198460 9838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:43:57.198621 9838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:43:57.212690 9838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:43:57.214576 9838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:43:57.216464 9838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:43:57.236250 9843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:57.236300 9838 server_base.cc:1034] running on GCE node
W20260928 10:43:57.236403 9846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:57.236577 9844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:57.237605 9838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:43:57.238844 9838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:43:57.240206 9838 hybrid_clock.cc:648] HybridClock initialized: now 1790592237240123 us; error 74 us; skew 500 ppm
I20260928 10:43:57.240595 9838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:43:57 dist-test-slave-mn38 krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592237, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:43:57.244311 9838 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:43:57.246490 9838 webserver.cc:460] Webserver started at http://127.8.218.194:46337/ using document root <none> and password file <none>
I20260928 10:43:57.247287 9838 fs_manager.cc:359] Metadata directory not provided
I20260928 10:43:57.247424 9838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:43:57.247942 9838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:43:57.251624 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "860f619619e54c9dbd220e074c76e143"
format_stamp: "Formatted at 2026-09-28 10:43:57 on dist-test-slave-mn38"
server_key: "50b9ab26e0da07ff9caa5d7026fceb56"
server_key_iv: "466240024df9f92aa628808a8969888f"
server_key_version: "encryptionkey@0"
I20260928 10:43:57.252790 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "860f619619e54c9dbd220e074c76e143"
format_stamp: "Formatted at 2026-09-28 10:43:57 on dist-test-slave-mn38"
server_key: "50b9ab26e0da07ff9caa5d7026fceb56"
server_key_iv: "466240024df9f92aa628808a8969888f"
server_key_version: "encryptionkey@0"
I20260928 10:43:57.259742 9838 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:43:57.264874 9853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:57.267419 9838 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:43:57.267709 9838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "860f619619e54c9dbd220e074c76e143"
format_stamp: "Formatted at 2026-09-28 10:43:57 on dist-test-slave-mn38"
server_key: "50b9ab26e0da07ff9caa5d7026fceb56"
server_key_iv: "466240024df9f92aa628808a8969888f"
server_key_version: "encryptionkey@0"
I20260928 10:43:57.267982 9838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:43:57.293794 9838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:43:57.298831 9838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:43:57.299147 9838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:43:57.300248 9838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:43:57.301787 9838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:43:57.301875 9838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:57.301992 9838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:43:57.302043 9838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:57.341380 9838 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:37305
I20260928 10:43:57.341562 9965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:37305 every 8 connection(s)
I20260928 10:43:57.342769 9838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:43:57.346464 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 9838
I20260928 10:43:57.346666 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:43:57.347131 9067 external_mini_cluster.cc:1428] Setting key 7a93810ccaf02dd5b680775a0cd6c17c
Sep 28 10:43:57 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592237, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:43:57.369897 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:57.346113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:41443 (local address 127.8.218.254:45215)
0928 10:43:57.346628 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:57.346635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:57.348365 (+ 1730us) server_negotiation.cc:407] Connection header received
0928 10:43:57.350099 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:57.350104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:57.350201 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:57.350371 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:57.352904 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.353793 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.356061 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.356779 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.356926 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:57.361424 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:57.361432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:57.361469 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:57.361516 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:57.365657 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:57.366897 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:57.366904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:57.366907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:57.367087 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:57.367741 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:57.367746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:57.367748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:57.368095 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:57.368240 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:57.369307 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0928 10:43:57.369580 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":258,"threads_started":1}
I20260928 10:43:57.371342 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:57.346727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45215 (local address 127.8.218.194:41443)
0928 10:43:57.347614 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:57.347674 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:57.349700 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:57.350683 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:57.350698 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:57.351539 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:43:57.352597 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:57.352630 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.354012 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.354018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:57.355823 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:57.355841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.357277 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.357290 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:57.357812 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:57.358712 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:43:57.358743 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:43:57.361052 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:43:57.365936 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:57.365948 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:57.365966 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:57.366717 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:57.367296 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:57.367301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:57.367304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:57.367548 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:57.368388 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:43:57.368399 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:43:57.368813 (+ 414us) client_negotiation.cc:769] Sending connection context
0928 10:43:57.369283 (+ 470us) client_negotiation.cc:240] Negotiation successful
0928 10:43:57.369991 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":300,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
I20260928 10:43:57.373533 9966 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45215
I20260928 10:43:57.374339 9966 heartbeater.cc:464] Registering TS with master...
I20260928 10:43:57.375748 9966 heartbeater.cc:511] Master 127.8.218.254:45215 requested a full tablet report, sending...
I20260928 10:43:57.378357 9643 ts_manager.cc:194] Registered new tserver with Master: 860f619619e54c9dbd220e074c76e143 (127.8.218.194:37305)
I20260928 10:43:57.379276 9643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:41443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:43:57.416842 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45215
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:43:57.790132 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:43:57.790601 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:43:57.790782 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:43:57.800958 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:43:57.801110 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:43:57.801236 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:43:57.813412 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:43:57.815372 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:43:57.817138 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:43:57.832056 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:57.832028 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:43:57.831992 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:43:57.833005 9972 server_base.cc:1034] running on GCE node
I20260928 10:43:57.833738 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:43:57.834856 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:43:57.835254 9832 heartbeater.cc:503] Master 127.8.218.254:45215 was elected leader, sending a full tablet report...
I20260928 10:43:57.836086 9972 hybrid_clock.cc:648] HybridClock initialized: now 1790592237836015 us; error 61 us; skew 500 ppm
I20260928 10:43:57.836500 9972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:43:57 dist-test-slave-mn38 krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592237, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:43:57.840003 9972 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:43:57.841641 9972 webserver.cc:460] Webserver started at http://127.8.218.195:36267/ using document root <none> and password file <none>
I20260928 10:43:57.842377 9972 fs_manager.cc:359] Metadata directory not provided
I20260928 10:43:57.842504 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:43:57.842814 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:43:57.846053 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "ce022a46315840819575136bd5795b08"
format_stamp: "Formatted at 2026-09-28 10:43:57 on dist-test-slave-mn38"
server_key: "ab654db79cd6884ef13de2fa1d78571f"
server_key_iv: "6d872cceb82537f30de7edfa7afa6f60"
server_key_version: "encryptionkey@0"
I20260928 10:43:57.846879 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "ce022a46315840819575136bd5795b08"
format_stamp: "Formatted at 2026-09-28 10:43:57 on dist-test-slave-mn38"
server_key: "ab654db79cd6884ef13de2fa1d78571f"
server_key_iv: "6d872cceb82537f30de7edfa7afa6f60"
server_key_version: "encryptionkey@0"
I20260928 10:43:57.852383 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 10:43:57.856705 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:57.858484 9972 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:43:57.858676 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "ce022a46315840819575136bd5795b08"
format_stamp: "Formatted at 2026-09-28 10:43:57 on dist-test-slave-mn38"
server_key: "ab654db79cd6884ef13de2fa1d78571f"
server_key_iv: "6d872cceb82537f30de7edfa7afa6f60"
server_key_version: "encryptionkey@0"
I20260928 10:43:57.858932 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:43:57.882597 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:43:57.887404 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:43:57.887670 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:43:57.888728 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:43:57.890409 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:43:57.890501 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:57.890591 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:43:57.890633 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:43:57.921334 9972 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:44915
I20260928 10:43:57.921416 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:44915 every 8 connection(s)
I20260928 10:43:57.922683 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:43:57.927137 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 9972
I20260928 10:43:57.927407 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:43:57.927829 9067 external_mini_cluster.cc:1428] Setting key 814f679db6fca264db17c8d037527d35
Sep 28 10:43:57 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592237, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:43:57.944571 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:57.925358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:34241 (local address 127.8.218.254:45215)
0928 10:43:57.925809 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:57.925814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:57.927005 (+ 1191us) server_negotiation.cc:407] Connection header received
0928 10:43:57.929030 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:57.929034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:57.929117 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:57.929244 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:57.931240 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.932002 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.933707 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.934110 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.934210 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:57.938337 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:57.938344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:57.938347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:57.938391 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:57.940988 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:57.942043 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:57.942048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:57.942051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:57.942243 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:57.942820 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:57.942824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:57.942826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:57.943167 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:57.943326 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:57.944121 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 10:43:57.944299 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":239,"threads_started":1}
I20260928 10:43:57.945578 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:57.925936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45215 (local address 127.8.218.195:34241)
0928 10:43:57.926805 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:57.926859 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:57.928754 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:57.929570 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:57.929583 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:57.930330 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:43:57.931034 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:57.931070 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.932181 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.932185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:57.933541 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:57.933557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.934330 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.934336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:57.934563 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:57.935423 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:43:57.935449 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:43:57.937965 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:43:57.941164 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:57.941174 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:57.941195 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:57.941895 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:57.942369 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:43:57.942374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:43:57.942377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:43:57.942639 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:43:57.943289 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:43:57.943296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:43:57.943719 (+ 423us) client_negotiation.cc:769] Sending connection context
0928 10:43:57.944006 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 10:43:57.944420 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":361,"threads_started":1}
I20260928 10:43:57.947202 10100 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45215
I20260928 10:43:57.947559 10100 heartbeater.cc:464] Registering TS with master...
I20260928 10:43:57.948329 10100 heartbeater.cc:511] Master 127.8.218.254:45215 requested a full tablet report, sending...
I20260928 10:43:57.950148 9643 ts_manager.cc:194] Registered new tserver with Master: ce022a46315840819575136bd5795b08 (127.8.218.195:44915)
I20260928 10:43:57.951115 9643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:34241
I20260928 10:43:57.957149 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:43:57.979065 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:57.964158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.8.218.254:45215)
0928 10:43:57.964320 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:57.964325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:57.964734 (+ 409us) server_negotiation.cc:407] Connection header received
0928 10:43:57.964916 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:57.964921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:57.965016 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:57.965181 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:57.966833 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.967584 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.968979 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:57.969401 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:57.969553 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:57.971935 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:57.971941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:57.971945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:57.971994 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:57.975233 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:57.976092 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:57.976098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:57.976102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:57.976263 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:57.976770 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:57.976775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:57.976777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:57.977221 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:57.977486 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:57.978404 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 10:43:57.978610 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 10:43:57.987643 9643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:43:57.996955 9643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:43:58.035856 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.012157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:44915 (local address 127.0.0.1:54530)
0928 10:43:58.023937 (+ 11780us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:58.023955 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:58.024176 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:58.026490 (+ 2314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:58.026499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:58.026544 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:43:58.027626 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.027641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.029770 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.029777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.032423 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.032439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.034637 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.034644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.034885 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.034949 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 10:43:58.035114 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 10:43:58.035368 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11654,"mutex_wait_us":392,"thread_start_us":11846,"threads_started":1}
I20260928 10:43:58.035854 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.012226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.8.218.193:44339)
0928 10:43:58.023477 (+ 11251us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.023487 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.023958 (+ 471us) server_negotiation.cc:407] Connection header received
0928 10:43:58.024742 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.024753 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.025124 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.025411 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:43:58.027454 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.030114 (+ 2660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.032830 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.034400 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.034532 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.034802 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.035372 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.035547 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10998,"thread_start_us":10819,"threads_started":1}
I20260928 10:43:58.035871 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.012431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.8.218.195:44915)
0928 10:43:58.025647 (+ 13216us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.025656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.025692 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:43:58.025804 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.025813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.026170 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.027080 (+ 910us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:43:58.027879 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.029574 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.032696 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.034389 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.034533 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.034794 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.035326 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.035517 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12911,"thread_start_us":13622,"threads_started":1}
I20260928 10:43:58.036684 10114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.012021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44339 (local address 127.0.0.1:58786)
0928 10:43:58.023692 (+ 11671us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:58.023772 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:58.024512 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:58.026490 (+ 1978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:58.026499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:58.026559 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:43:58.027213 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.027234 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.030349 (+ 3115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.030355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.032543 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.032562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.034951 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.034960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.035145 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.035174 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 10:43:58.035252 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 10:43:58.036289 (+ 1037us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11338,"mutex_wait_us":968,"spinlock_wait_cycles":11904,"thread_start_us":11630,"threads_started":1}
I20260928 10:43:58.039283 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.027184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:37305 (local address 127.0.0.1:52660)
0928 10:43:58.030388 (+ 3204us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:58.030409 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:58.030577 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:58.031467 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:58.031478 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:58.031518 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:43:58.032037 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.032052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.034189 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.034194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.036205 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.036219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.038412 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.038419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.038599 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.038635 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:43:58.038777 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 10:43:58.038927 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3032,"thread_start_us":2819,"threads_started":1}
I20260928 10:43:58.039356 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.025254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.8.218.194:37305)
0928 10:43:58.029808 (+ 4554us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.029842 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.030548 (+ 706us) server_negotiation.cc:407] Connection header received
0928 10:43:58.030796 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.030806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.031148 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.031459 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:43:58.032275 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.034024 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.036375 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.037869 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.038015 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.038255 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.038863 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.039051 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2460,"thread_start_us":2306,"threads_started":1}
I20260928 10:43:58.040920 10035 tablet_service.cc:1467] Processing CreateTablet for tablet 4a23bcddbac7425e836868159fe4d496 (DEFAULT_TABLE table=test-table [id=2b8e881101034e8482f8cc2672c9111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:43:58.040927 9767 tablet_service.cc:1467] Processing CreateTablet for tablet 4a23bcddbac7425e836868159fe4d496 (DEFAULT_TABLE table=test-table [id=2b8e881101034e8482f8cc2672c9111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:43:58.042308 10035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a23bcddbac7425e836868159fe4d496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:43:58.042354 9767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a23bcddbac7425e836868159fe4d496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:43:58.043567 9901 tablet_service.cc:1467] Processing CreateTablet for tablet 4a23bcddbac7425e836868159fe4d496 (DEFAULT_TABLE table=test-table [id=2b8e881101034e8482f8cc2672c9111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:43:58.045001 9901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a23bcddbac7425e836868159fe4d496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:43:58.054006 10119 tablet_bootstrap.cc:492] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08: Bootstrap starting.
I20260928 10:43:58.057009 10120 tablet_bootstrap.cc:492] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49: Bootstrap starting.
I20260928 10:43:58.058995 10119 tablet_bootstrap.cc:654] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08: Neither blocks nor log segments found. Creating new log.
I20260928 10:43:58.059782 10121 tablet_bootstrap.cc:492] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143: Bootstrap starting.
I20260928 10:43:58.060434 10119 log.cc:824] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08: Log is configured to *not* fsync() on all Append() calls
I20260928 10:43:58.061221 10120 tablet_bootstrap.cc:654] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49: Neither blocks nor log segments found. Creating new log.
I20260928 10:43:58.062407 10120 log.cc:824] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49: Log is configured to *not* fsync() on all Append() calls
I20260928 10:43:58.063266 10121 tablet_bootstrap.cc:654] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143: Neither blocks nor log segments found. Creating new log.
I20260928 10:43:58.064033 10119 tablet_bootstrap.cc:492] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08: No bootstrap required, opened a new log
I20260928 10:43:58.064358 10119 ts_tablet_manager.cc:1397] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 10:43:58.064507 10121 log.cc:824] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143: Log is configured to *not* fsync() on all Append() calls
I20260928 10:43:58.065920 10120 tablet_bootstrap.cc:492] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49: No bootstrap required, opened a new log
I20260928 10:43:58.066289 10120 ts_tablet_manager.cc:1397] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 10:43:58.067406 10121 tablet_bootstrap.cc:492] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143: No bootstrap required, opened a new log
I20260928 10:43:58.067658 10121 ts_tablet_manager.cc:1397] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 10:43:58.075476 10121 raft_consensus.cc:348] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
I20260928 10:43:58.075922 10121 raft_consensus.cc:374] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:43:58.076042 10121 raft_consensus.cc:729] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860f619619e54c9dbd220e074c76e143, State: Initialized, Role: FOLLOWER
I20260928 10:43:58.076104 10119 raft_consensus.cc:348] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
I20260928 10:43:58.076573 10119 raft_consensus.cc:374] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:43:58.076686 10119 raft_consensus.cc:729] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce022a46315840819575136bd5795b08, State: Initialized, Role: FOLLOWER
I20260928 10:43:58.076741 10121 consensus_queue.cc:260] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
I20260928 10:43:58.077318 10120 raft_consensus.cc:348] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
I20260928 10:43:58.077920 10120 raft_consensus.cc:374] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:43:58.078027 10120 raft_consensus.cc:729] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9665104697c3493eb909a1108407fc49, State: Initialized, Role: FOLLOWER
I20260928 10:43:58.077556 10119 consensus_queue.cc:260] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
I20260928 10:43:58.078783 10120 consensus_queue.cc:260] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
I20260928 10:43:58.079722 10100 heartbeater.cc:503] Master 127.8.218.254:45215 was elected leader, sending a full tablet report...
I20260928 10:43:58.080628 10119 ts_tablet_manager.cc:1428] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260928 10:43:58.081897 9966 heartbeater.cc:503] Master 127.8.218.254:45215 was elected leader, sending a full tablet report...
I20260928 10:43:58.083449 10120 ts_tablet_manager.cc:1428] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260928 10:43:58.084667 10121 ts_tablet_manager.cc:1428] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260928 10:43:58.095000 9967 tablet.cc:2369] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:43:58.173080 10125 raft_consensus.cc:482] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:43:58.173396 10125 raft_consensus.cc:504] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
W20260928 10:43:58.174357 10101 tablet.cc:2369] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:43:58.175367 10125 leader_election.cc:288] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9665104697c3493eb909a1108407fc49 (127.8.218.193:44339), ce022a46315840819575136bd5795b08 (127.8.218.195:44915)
I20260928 10:43:58.182987 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.175906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:58495 (local address 127.8.218.195:44915)
0928 10:43:58.176102 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.176108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.176925 (+ 817us) server_negotiation.cc:407] Connection header received
0928 10:43:58.177082 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.177086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.177146 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.177224 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:43:58.177976 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.178751 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.180897 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.181964 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.182132 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.182210 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.182584 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.182720 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:43:58.182993 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.175837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:46609 (local address 127.8.218.193:44339)
0928 10:43:58.176060 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.176066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.176797 (+ 731us) server_negotiation.cc:407] Connection header received
0928 10:43:58.177002 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.177006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.177071 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.177150 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:43:58.177989 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.178739 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.180836 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.181864 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.182026 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.182111 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.182560 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.182714 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 10:43:58.183151 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.175867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44339 (local address 127.8.218.194:46609)
0928 10:43:58.176665 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:58.176685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:58.176866 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:58.177180 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:58.177185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:58.177214 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:43:58.177739 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.177758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.178889 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.178895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.180609 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.180632 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.182126 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.182134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.182334 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.182394 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 10:43:58.182505 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 10:43:58.182666 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":25,"spinlock_wait_cycles":29568,"thread_start_us":465,"threads_started":1}
I20260928 10:43:58.183748 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.175956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:44915 (local address 127.8.218.194:58495)
0928 10:43:58.176800 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 10:43:58.176815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:43:58.176949 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:43:58.177249 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:43:58.177253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:43:58.177281 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:43:58.177811 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.177826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.178894 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.178900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.180674 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:43:58.180690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.182208 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.182215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:43:58.182408 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.182433 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:43:58.183266 (+ 833us) client_negotiation.cc:240] Negotiation successful
0928 10:43:58.183430 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":159,"thread_start_us":417,"threads_started":1}
I20260928 10:43:58.184276 9787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23bcddbac7425e836868159fe4d496" candidate_uuid: "860f619619e54c9dbd220e074c76e143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9665104697c3493eb909a1108407fc49" is_pre_election: true
I20260928 10:43:58.184847 9787 raft_consensus.cc:2454] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860f619619e54c9dbd220e074c76e143 in term 0.
I20260928 10:43:58.184911 10055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23bcddbac7425e836868159fe4d496" candidate_uuid: "860f619619e54c9dbd220e074c76e143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce022a46315840819575136bd5795b08" is_pre_election: true
I20260928 10:43:58.185604 10055 raft_consensus.cc:2454] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860f619619e54c9dbd220e074c76e143 in term 0.
I20260928 10:43:58.185626 9857 leader_election.cc:302] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 860f619619e54c9dbd220e074c76e143, 9665104697c3493eb909a1108407fc49; no voters:
I20260928 10:43:58.186029 10125 raft_consensus.cc:2781] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:43:58.186180 10125 raft_consensus.cc:482] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:43:58.186296 10125 raft_consensus.cc:3037] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:43:58.188585 10125 raft_consensus.cc:504] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
I20260928 10:43:58.189257 10125 leader_election.cc:288] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [CANDIDATE]: Term 1 election: Requested vote from peers 9665104697c3493eb909a1108407fc49 (127.8.218.193:44339), ce022a46315840819575136bd5795b08 (127.8.218.195:44915)
I20260928 10:43:58.189935 10055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23bcddbac7425e836868159fe4d496" candidate_uuid: "860f619619e54c9dbd220e074c76e143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce022a46315840819575136bd5795b08"
I20260928 10:43:58.189960 9787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a23bcddbac7425e836868159fe4d496" candidate_uuid: "860f619619e54c9dbd220e074c76e143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9665104697c3493eb909a1108407fc49"
I20260928 10:43:58.190147 10055 raft_consensus.cc:3037] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:43:58.190177 9787 raft_consensus.cc:3037] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:43:58.192647 9787 raft_consensus.cc:2454] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860f619619e54c9dbd220e074c76e143 in term 1.
I20260928 10:43:58.192643 10055 raft_consensus.cc:2454] T 4a23bcddbac7425e836868159fe4d496 P ce022a46315840819575136bd5795b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860f619619e54c9dbd220e074c76e143 in term 1.
I20260928 10:43:58.193279 9855 leader_election.cc:302] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 860f619619e54c9dbd220e074c76e143, ce022a46315840819575136bd5795b08; no voters:
I20260928 10:43:58.193658 10125 raft_consensus.cc:2781] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:43:58.194104 10125 raft_consensus.cc:686] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [term 1 LEADER]: Becoming Leader. State: Replica: 860f619619e54c9dbd220e074c76e143, State: Running, Role: LEADER
I20260928 10:43:58.194692 10125 consensus_queue.cc:237] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } }
I20260928 10:43:58.200351 9643 catalog_manager.cc:5495] T 4a23bcddbac7425e836868159fe4d496 P 860f619619e54c9dbd220e074c76e143 reported cstate change: term changed from 0 to 1, leader changed from <none> to 860f619619e54c9dbd220e074c76e143 (127.8.218.194). New cstate: current_term: 1 leader_uuid: "860f619619e54c9dbd220e074c76e143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9665104697c3493eb909a1108407fc49" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce022a46315840819575136bd5795b08" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "860f619619e54c9dbd220e074c76e143" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37305 } health_report { overall_health: HEALTHY } } }
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:43:58.240552 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.223009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.8.218.193:44339)
0928 10:43:58.223274 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.223281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.223605 (+ 324us) server_negotiation.cc:407] Connection header received
0928 10:43:58.223825 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.223829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.223921 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.224025 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:58.225724 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.226634 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.228344 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.228805 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.228956 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.233617 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:58.233635 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:58.233652 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:58.233721 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:58.236990 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.237982 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.237993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.237996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.238170 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.238691 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.238698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.238702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.239224 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:58.239514 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.239982 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.240177 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:43:58.287220 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.271488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.8.218.193:44339)
0928 10:43:58.271726 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.271732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.272223 (+ 491us) server_negotiation.cc:407] Connection header received
0928 10:43:58.272438 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.272442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.272514 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.272606 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:58.274118 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.274924 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.276330 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.276788 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.276905 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.281254 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:58.281261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:58.281265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:58.281315 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:58.284189 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.285039 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.285044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.285046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.285218 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.285767 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.285770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.285772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.286077 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:58.286290 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.286692 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.286903 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 10:43:58.300334 9833 tablet.cc:2369] T 4a23bcddbac7425e836868159fe4d496 P 9665104697c3493eb909a1108407fc49: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[9598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:43:58.335757 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.320493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.8.218.193:44339)
0928 10:43:58.320711 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.320716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.320838 (+ 122us) server_negotiation.cc:407] Connection header received
0928 10:43:58.320995 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.321000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.321069 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.321162 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:58.322825 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.323632 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.325194 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.325751 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.325870 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.330379 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:58.330385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:58.330389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:58.330433 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:58.333030 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.333919 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.333924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.333926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.334068 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.334444 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.334448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.334450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.334773 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:58.334967 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.335229 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.335471 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 10:43:58.337523 9747 tablet_service.cc:2908] Rejecting scan request for tablet 4a23bcddbac7425e836868159fe4d496: Uninitialized: clean time has not yet been initialized
I20260928 10:43:58.340648 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 9704
I20260928 10:43:58.355365 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.341915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.8.218.193:44339)
0928 10:43:58.342113 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.342121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.342499 (+ 378us) server_negotiation.cc:407] Connection header received
0928 10:43:58.342642 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.342647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.342730 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.342842 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:58.344047 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.345088 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.346504 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.347130 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.347270 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.349523 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:58.349530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:58.349535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:58.349579 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:58.352697 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.353479 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.353485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.353488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.353630 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.353986 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.353991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.353994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.354365 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:58.354636 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.354822 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.355041 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:43:58.356029 9827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:43:58.445870 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 9704
I20260928 10:43:58.460078 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 9838
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:43:58.475899 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.460920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.8.218.194:37305)
0928 10:43:58.461093 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.461098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.461116 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:43:58.461233 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.461237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.461339 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.461422 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:58.462691 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.463415 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.464788 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.465193 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.465297 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.469678 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:58.469695 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:58.469713 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:58.469766 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:58.472861 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.473695 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.473703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.473705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.473911 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.474358 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.474362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.474364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.474811 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:58.475039 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.475366 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.475553 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 10:43:58.476641 9961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:43:58.568284 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 9838
I20260928 10:43:58.582556 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 9972
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[9598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:43:58.598737 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:43:58.583297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.8.218.195:44915)
0928 10:43:58.583496 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:43:58.583502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:43:58.583523 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:43:58.583719 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:43:58.583724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:43:58.583796 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:43:58.583882 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:43:58.585187 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.585952 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.587255 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:43:58.587677 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:43:58.587799 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:43:58.592580 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:43:58.592600 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:43:58.592617 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:43:58.592662 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:43:58.595614 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.596437 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.596446 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.596448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.596595 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:43:58.596998 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:43:58.597001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:43:58.597003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:43:58.597393 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:43:58.597667 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 10:43:58.598066 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 10:43:58.598275 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:43:58.599498 10095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:43:58.685834 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 9972
I20260928 10:43:58.699216 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 9611
I20260928 10:43:58.700387 9673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:43:58.811892 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 9611
2026-09-28T10:43:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4324 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[10149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[10149](info): set up 2 sockets
Sep 28 10:43:58 dist-test-slave-mn38 krb5kdc[10149](info): commencing operation
krb5kdc: starting...
W20260928 10:43:59.818575 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:43:59 dist-test-slave-mn38 krb5kdc[10149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:44:00.010612 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:34335
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:00.363020 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:00.363345 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:00.363461 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:00.371021 10162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:00.371161 10162 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:44:00.371204 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:00.371294 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:00.371361 10162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:00.371443 10162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:00.382256 10162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:34335
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:00.383723 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:00.385531 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:00.399338 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:00.399338 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:00.399544 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:00.400048 10162 server_base.cc:1034] running on GCE node
I20260928 10:44:00.401051 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:00.402818 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:00.404068 10162 hybrid_clock.cc:648] HybridClock initialized: now 1790592240403952 us; error 93 us; skew 500 ppm
I20260928 10:44:00.404534 10162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:00 dist-test-slave-mn38 krb5kdc[10149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592240, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:00.408605 10162 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:44:00.410254 10162 webserver.cc:460] Webserver started at http://127.8.218.254:35771/ using document root <none> and password file <none>
I20260928 10:44:00.411000 10162 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:00.411120 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:00.411442 10162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:00.414702 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "0cb8343ac0ac4048a7829e0760364f7a"
format_stamp: "Formatted at 2026-09-28 10:44:00 on dist-test-slave-mn38"
server_key: "4fe5d0c1543b49561da4e92943f2564e"
server_key_iv: "45d18ff4f8e83cf3a9a46803bd8b110e"
server_key_version: "encryptionkey@0"
I20260928 10:44:00.415577 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "0cb8343ac0ac4048a7829e0760364f7a"
format_stamp: "Formatted at 2026-09-28 10:44:00 on dist-test-slave-mn38"
server_key: "4fe5d0c1543b49561da4e92943f2564e"
server_key_iv: "45d18ff4f8e83cf3a9a46803bd8b110e"
server_key_version: "encryptionkey@0"
I20260928 10:44:00.421676 10162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:44:00.425926 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:00.428292 10162 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:44:00.428521 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "0cb8343ac0ac4048a7829e0760364f7a"
format_stamp: "Formatted at 2026-09-28 10:44:00 on dist-test-slave-mn38"
server_key: "4fe5d0c1543b49561da4e92943f2564e"
server_key_iv: "45d18ff4f8e83cf3a9a46803bd8b110e"
server_key_version: "encryptionkey@0"
I20260928 10:44:00.428783 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:00.467592 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:00.473225 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:00.473582 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:00.497466 10162 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:34335
I20260928 10:44:00.497534 10238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:34335 every 8 connection(s)
I20260928 10:44:00.499624 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:00.500645 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 10162
I20260928 10:44:00.500898 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:00.501773 9067 external_mini_cluster.cc:1428] Setting key 65cffaeb7e11637c378ec30369d87c64
I20260928 10:44:00.505223 10239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:44:00 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:00.521795 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a: Bootstrap starting.
I20260928 10:44:00.526309 10239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:00.527757 10239 log.cc:824] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:00.530867 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:00.504128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.8.218.254:34335)
0928 10:44:00.504945 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:00.504957 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:00.505002 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:44:00.507036 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:00.507065 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:00.507573 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:00.508115 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:00.510404 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:00.512335 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:00.514595 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:00.515554 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:00.515798 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:00.521593 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:00.521608 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:00.521639 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:00.521698 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:00.525416 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:00.526591 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:00.526602 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:00.526611 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:00.526840 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:00.527322 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:00.527327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:00.527329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:00.528188 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:00.528544 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:00.529214 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 10:44:00.529737 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":368,"threads_started":1}
I20260928 10:44:00.531450 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a: No bootstrap required, opened a new log
I20260928 10:44:00.539655 10239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8343ac0ac4048a7829e0760364f7a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34335 } }
I20260928 10:44:00.540076 10239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:00.540184 10239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb8343ac0ac4048a7829e0760364f7a, State: Initialized, Role: FOLLOWER
I20260928 10:44:00.540819 10239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8343ac0ac4048a7829e0760364f7a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34335 } }
I20260928 10:44:00.541059 10239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:00.541205 10239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:00.541364 10239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:00.543691 10239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8343ac0ac4048a7829e0760364f7a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34335 } }
I20260928 10:44:00.544232 10239 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cb8343ac0ac4048a7829e0760364f7a; no voters:
I20260928 10:44:00.544823 10239 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:00.544955 10244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:00.545636 10244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [term 1 LEADER]: Becoming Leader. State: Replica: 0cb8343ac0ac4048a7829e0760364f7a, State: Running, Role: LEADER
I20260928 10:44:00.546221 10244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8343ac0ac4048a7829e0760364f7a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34335 } }
I20260928 10:44:00.546922 10239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:00.549188 10245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cb8343ac0ac4048a7829e0760364f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8343ac0ac4048a7829e0760364f7a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34335 } } }
I20260928 10:44:00.549825 10245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:00.549160 10246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb8343ac0ac4048a7829e0760364f7a. Latest consensus state: current_term: 1 leader_uuid: "0cb8343ac0ac4048a7829e0760364f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8343ac0ac4048a7829e0760364f7a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34335 } } }
I20260928 10:44:00.550436 10246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:00.557348 10251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:00.563441 10251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:00.574621 10251 catalog_manager.cc:1347] Generated new cluster ID: b2155901e72544f091ef689e5a5aa10a
I20260928 10:44:00.574774 10251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:00.589399 10251 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:00.591450 10251 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:00.609189 10251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a: Generated new TSK 0
I20260928 10:44:00.610558 10251 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:44:00.674372 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:34335
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:01.120786 10265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:01.121094 10265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:01.121191 10265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:01.128569 10265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:01.128700 10265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:01.128786 10265 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:01.128866 10265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:01.139185 10265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:01.140574 10265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:01.142040 10265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:01.156651 10270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:01.156690 10273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:01.156620 10271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:01.157877 10265 server_base.cc:1034] running on GCE node
I20260928 10:44:01.158583 10265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:01.159679 10265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:01.160998 10265 hybrid_clock.cc:648] HybridClock initialized: now 1790592241160888 us; error 76 us; skew 500 ppm
I20260928 10:44:01.161758 10265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:01 dist-test-slave-mn38 krb5kdc[10149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592241, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:01.168776 10265 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:44:01.170605 10265 webserver.cc:460] Webserver started at http://127.8.218.193:37045/ using document root <none> and password file <none>
I20260928 10:44:01.171480 10265 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:01.171603 10265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:01.171972 10265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:01.175282 10265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "7ab041a929744592828a1b43ab45802f"
format_stamp: "Formatted at 2026-09-28 10:44:01 on dist-test-slave-mn38"
server_key: "076f2ac3be18c6bdf728c94a2df0f869"
server_key_iv: "0de7b84cf83ad3d25eebdb62f60504af"
server_key_version: "encryptionkey@0"
I20260928 10:44:01.176249 10265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "7ab041a929744592828a1b43ab45802f"
format_stamp: "Formatted at 2026-09-28 10:44:01 on dist-test-slave-mn38"
server_key: "076f2ac3be18c6bdf728c94a2df0f869"
server_key_iv: "0de7b84cf83ad3d25eebdb62f60504af"
server_key_version: "encryptionkey@0"
I20260928 10:44:01.182757 10265 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:44:01.187665 10280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:01.189656 10265 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:44:01.189879 10265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "7ab041a929744592828a1b43ab45802f"
format_stamp: "Formatted at 2026-09-28 10:44:01 on dist-test-slave-mn38"
server_key: "076f2ac3be18c6bdf728c94a2df0f869"
server_key_iv: "0de7b84cf83ad3d25eebdb62f60504af"
server_key_version: "encryptionkey@0"
I20260928 10:44:01.190138 10265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:01.209892 10265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:01.215022 10265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:01.215315 10265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:01.218539 10265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:01.218673 10265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:01.218811 10265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:01.218879 10265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:01 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592241, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:01.245020 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:01.219223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:58147 (local address 127.8.218.254:34335)
0928 10:44:01.219778 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:01.219783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:01.221213 (+ 1430us) server_negotiation.cc:407] Connection header received
0928 10:44:01.223559 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:01.223566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:01.223667 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:01.223781 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:01.227060 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.228155 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.230768 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.231355 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.231486 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:01.237555 (+ 6069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:01.237564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:01.237569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:01.237624 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:01.241203 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:01.242366 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:01.242372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:01.242374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:01.242545 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:01.243141 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:01.243145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:01.243147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:01.243436 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:01.243582 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:01.244368 (+ 786us) server_negotiation.cc:299] Negotiation successful
0928 10:44:01.244637 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":354,"threads_started":1}
I20260928 10:44:01.246455 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:01.220004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:34335 (local address 127.8.218.193:58147)
0928 10:44:01.220974 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:01.221029 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:01.223164 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:01.224419 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:01.224436 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:01.225522 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:01.226745 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:01.226791 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.228394 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.228398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:01.230553 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:01.230579 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.231706 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.231715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:01.232147 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:01.233115 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:01.233154 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:01.237110 (+ 3956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:01.241406 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:01.241422 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:01.241480 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:01.242191 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:01.242716 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:01.242721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:01.242729 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:01.242993 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:01.243574 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:01.243583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:01.243977 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 10:44:01.244242 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 10:44:01.244821 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":404,"threads_started":1}
I20260928 10:44:01.271179 10265 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:35735
I20260928 10:44:01.271188 10400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:35735 every 8 connection(s)
I20260928 10:44:01.272832 10265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:01.273228 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:01.253093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:50887 (local address 127.8.218.254:34335)
0928 10:44:01.253274 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:01.253281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:01.258546 (+ 5265us) server_negotiation.cc:407] Connection header received
0928 10:44:01.258766 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:01.258770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:01.258840 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:01.258921 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:01.260053 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.260772 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.262813 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.263532 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.263693 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:01.266326 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:01.266334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:01.266339 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:01.266390 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:01.269859 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:01.270863 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:01.270867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:01.270870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:01.271023 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:01.271482 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:01.271486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:01.271489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:01.271798 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:01.271962 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:01.272532 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 10:44:01.272834 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:44:01.273413 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:01.253067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:34335 (local address 127.8.218.193:50887)
0928 10:44:01.258370 (+ 5303us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:01.258408 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:01.258617 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:01.258983 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:01.258989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:01.259532 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:01.259879 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:01.259889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.260941 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.260948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:01.262558 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:01.262575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.263848 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.263857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:01.263992 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:01.264747 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:01.264767 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:01.265708 (+ 941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:01.270102 (+ 4394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:01.270109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:01.270114 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:01.270702 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:01.271162 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:01.271168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:01.271176 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:01.271359 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:01.271952 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:01.271957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:01.272167 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 10:44:01.272294 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 10:44:01.272872 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4903,"thread_start_us":4658,"threads_started":1}
I20260928 10:44:01.276124 10401 heartbeater.cc:347] Connected to a master server at 127.8.218.254:34335
I20260928 10:44:01.276419 10401 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:01.277611 10401 heartbeater.cc:511] Master 127.8.218.254:34335 requested a full tablet report, sending...
I20260928 10:44:01.279173 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 10265
I20260928 10:44:01.279364 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:01.279980 9067 external_mini_cluster.cc:1428] Setting key 2d4500e99432ec97dd02e36007dad243
I20260928 10:44:01.282611 10194 ts_manager.cc:194] Registered new tserver with Master: 7ab041a929744592828a1b43ab45802f (127.8.218.193:35735)
I20260928 10:44:01.284264 10194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:58147
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:44:01.325311 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:34335
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:01.702049 10405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:01.702409 10405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:01.702546 10405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:01.710638 10405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:01.710805 10405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:01.710929 10405 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:01.711046 10405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:01.723531 10405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:01.725078 10405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:01.726796 10405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:01.743218 10411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:01.743296 10413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:01.743297 10410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:01.744156 10405 server_base.cc:1034] running on GCE node
I20260928 10:44:01.745293 10405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:01.746812 10405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:01.748132 10405 hybrid_clock.cc:648] HybridClock initialized: now 1790592241748051 us; error 59 us; skew 500 ppm
I20260928 10:44:01.748509 10405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:01 dist-test-slave-mn38 krb5kdc[10149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592241, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:01.753096 10405 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:44:01.755026 10405 webserver.cc:460] Webserver started at http://127.8.218.194:34387/ using document root <none> and password file <none>
I20260928 10:44:01.755795 10405 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:01.755918 10405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:01.756256 10405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:01.759519 10405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "1908143790fe4a65acd0aa6fbc3fdb0d"
format_stamp: "Formatted at 2026-09-28 10:44:01 on dist-test-slave-mn38"
server_key: "fe5fac4a4d3ba841ebf8d75bdb9fef51"
server_key_iv: "4674d28075946b3cd52fa798023aab8f"
server_key_version: "encryptionkey@0"
I20260928 10:44:01.760424 10405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "1908143790fe4a65acd0aa6fbc3fdb0d"
format_stamp: "Formatted at 2026-09-28 10:44:01 on dist-test-slave-mn38"
server_key: "fe5fac4a4d3ba841ebf8d75bdb9fef51"
server_key_iv: "4674d28075946b3cd52fa798023aab8f"
server_key_version: "encryptionkey@0"
I20260928 10:44:01.766724 10405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:44:01.772100 10420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:01.774314 10405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:44:01.774530 10405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "1908143790fe4a65acd0aa6fbc3fdb0d"
format_stamp: "Formatted at 2026-09-28 10:44:01 on dist-test-slave-mn38"
server_key: "fe5fac4a4d3ba841ebf8d75bdb9fef51"
server_key_iv: "4674d28075946b3cd52fa798023aab8f"
server_key_version: "encryptionkey@0"
I20260928 10:44:01.774780 10405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:01.799497 10405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:01.804586 10405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:01.804888 10405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:01.807878 10405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:01.807994 10405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:01.808148 10405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:01.808209 10405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:01 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592241, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:01.836303 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:01.808831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:47093 (local address 127.8.218.254:34335)
0928 10:44:01.809319 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:01.809326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:01.810913 (+ 1587us) server_negotiation.cc:407] Connection header received
0928 10:44:01.814339 (+ 3426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:01.814345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:01.814468 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:01.814610 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:01.818342 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.819519 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.821800 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.822432 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.822593 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:01.828777 (+ 6184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:01.828786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:01.828792 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:01.828847 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:01.832276 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:01.833176 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:01.833181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:01.833183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:01.833318 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:01.833997 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:01.834003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:01.834006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:01.834340 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:01.834527 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:01.835778 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0928 10:44:01.836014 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
I20260928 10:44:01.838279 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:01.809672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:34335 (local address 127.8.218.194:47093)
0928 10:44:01.810664 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:01.810735 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:01.813842 (+ 3107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:01.815022 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:01.815039 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:01.816602 (+ 1563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:01.817950 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:01.817997 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.819821 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.819827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:01.821573 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:01.821598 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.822793 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.822803 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:01.823453 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:01.824664 (+ 1211us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:01.824715 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:01.828402 (+ 3687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:01.832446 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:01.832459 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:01.832482 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:01.833028 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:01.833476 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:01.833483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:01.833488 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:01.833838 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:01.834502 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:01.834513 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:01.835188 (+ 675us) client_negotiation.cc:769] Sending connection context
0928 10:44:01.835637 (+ 449us) client_negotiation.cc:240] Negotiation successful
0928 10:44:01.836349 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":353,"threads_started":1}
I20260928 10:44:01.858501 10405 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:35617
I20260928 10:44:01.858511 10540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:35617 every 8 connection(s)
I20260928 10:44:01.860152 10405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:01.860603 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:01.844543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:34335 (local address 127.8.218.194:37549)
0928 10:44:01.845503 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:01.845523 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:01.845701 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:01.846032 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:01.846036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:01.846958 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:01.847378 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:01.847397 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.849006 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.849011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:01.850456 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:01.850477 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.851640 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.851649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:01.851900 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:01.852717 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:01.852748 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:01.853959 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:01.857757 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:01.857763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:01.857767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:01.858270 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:01.858713 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:01.858719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:01.858723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:01.858855 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:01.859638 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:01.859642 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:01.859814 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 10:44:01.859932 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 10:44:01.860225 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":621,"threads_started":1}
I20260928 10:44:01.860667 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:01.844618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:37549 (local address 127.8.218.254:34335)
0928 10:44:01.844860 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:01.844868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:01.845628 (+ 760us) server_negotiation.cc:407] Connection header received
0928 10:44:01.845825 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:01.845829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:01.845896 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:01.845977 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:01.847670 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.848753 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.850680 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:01.851385 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:01.851556 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:01.854301 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:01.854306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:01.854310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:01.854345 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:01.857530 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:01.858427 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:01.858434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:01.858438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:01.858570 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:01.859001 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:01.859007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:01.859010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:01.859493 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:01.859657 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:01.860073 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 10:44:01.860336 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:44:01.862707 10541 heartbeater.cc:347] Connected to a master server at 127.8.218.254:34335
I20260928 10:44:01.863613 10541 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:01.864483 10541 heartbeater.cc:511] Master 127.8.218.254:34335 requested a full tablet report, sending...
I20260928 10:44:01.866460 10194 ts_manager.cc:194] Registered new tserver with Master: 1908143790fe4a65acd0aa6fbc3fdb0d (127.8.218.194:35617)
I20260928 10:44:01.867300 10194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:47093
I20260928 10:44:01.867398 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 10405
I20260928 10:44:01.867583 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:01.868007 9067 external_mini_cluster.cc:1428] Setting key d47586606711826bc1d2fd71f1b5c57b
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:44:01.934213 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:34335
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:44:02.288350 10401 heartbeater.cc:503] Master 127.8.218.254:34335 was elected leader, sending a full tablet report...
W20260928 10:44:02.296360 10545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:02.296690 10545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:02.296816 10545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:02.304687 10545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:02.304833 10545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:02.304940 10545 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:02.305040 10545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:02.315788 10545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:02.317272 10545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:02.318913 10545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:02.334677 10550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:02.334645 10551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:02.334609 10553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:02.335618 10545 server_base.cc:1034] running on GCE node
I20260928 10:44:02.336309 10545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:02.337371 10545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:02.338676 10545 hybrid_clock.cc:648] HybridClock initialized: now 1790592242338601 us; error 47 us; skew 500 ppm
I20260928 10:44:02.339002 10545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:02 dist-test-slave-mn38 krb5kdc[10149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592242, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:02.342281 10545 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:44:02.343801 10545 webserver.cc:460] Webserver started at http://127.8.218.195:37671/ using document root <none> and password file <none>
I20260928 10:44:02.344502 10545 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:02.344594 10545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:02.344867 10545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:02.348320 10545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "66aaa86bb5e6468a96f4aec4a5fa219c"
format_stamp: "Formatted at 2026-09-28 10:44:02 on dist-test-slave-mn38"
server_key: "40d71c6b551adb994b2bf04b46db466a"
server_key_iv: "3cc4fbeb978ae18c48974eba68c9862f"
server_key_version: "encryptionkey@0"
I20260928 10:44:02.349272 10545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "66aaa86bb5e6468a96f4aec4a5fa219c"
format_stamp: "Formatted at 2026-09-28 10:44:02 on dist-test-slave-mn38"
server_key: "40d71c6b551adb994b2bf04b46db466a"
server_key_iv: "3cc4fbeb978ae18c48974eba68c9862f"
server_key_version: "encryptionkey@0"
I20260928 10:44:02.355218 10545 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 10:44:02.359655 10560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:02.361405 10545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:44:02.361624 10545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "66aaa86bb5e6468a96f4aec4a5fa219c"
format_stamp: "Formatted at 2026-09-28 10:44:02 on dist-test-slave-mn38"
server_key: "40d71c6b551adb994b2bf04b46db466a"
server_key_iv: "3cc4fbeb978ae18c48974eba68c9862f"
server_key_version: "encryptionkey@0"
I20260928 10:44:02.361835 10545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:02.395828 10545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:02.401774 10545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:02.402087 10545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:02.405397 10545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:02.405548 10545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:44:02.405691 10545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:02.405751 10545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:02 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592242, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:02.432339 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.406236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:60049 (local address 127.8.218.254:34335)
0928 10:44:02.406686 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.406692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.407975 (+ 1283us) server_negotiation.cc:407] Connection header received
0928 10:44:02.411109 (+ 3134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.411116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.411221 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.411347 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:02.414294 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.415537 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.418183 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.418791 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.418915 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.424106 (+ 5191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:02.424112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:02.424115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:02.424155 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:02.427636 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:02.429016 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:02.429022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:02.429026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:02.429225 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:02.429850 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:02.429856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:02.429860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:02.430291 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:02.430484 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.431648 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.431927 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260928 10:44:02.433682 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.406847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:34335 (local address 127.8.218.195:60049)
0928 10:44:02.407738 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:02.407800 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:02.410567 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:02.411872 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:02.411886 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:02.412739 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:02.413994 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.414025 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.415772 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.415779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.417893 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.417918 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.419072 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.419082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.419307 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.420106 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:02.420143 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:02.423655 (+ 3512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:02.427858 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:02.427873 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:02.427898 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:02.428782 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:02.429380 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:02.429389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:02.429393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:02.429696 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:02.430496 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:02.430506 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:02.431119 (+ 613us) client_negotiation.cc:769] Sending connection context
0928 10:44:02.431518 (+ 399us) client_negotiation.cc:240] Negotiation successful
0928 10:44:02.432187 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":329,"threads_started":1}
I20260928 10:44:02.455741 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.439580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:35995 (local address 127.8.218.254:34335)
0928 10:44:02.439778 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.439791 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.440622 (+ 831us) server_negotiation.cc:407] Connection header received
0928 10:44:02.440808 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.440813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.440918 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.441020 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:02.442335 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.443073 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.445153 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.445775 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.445917 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.449259 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:02.449265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:02.449269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:02.449317 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:02.452490 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:02.453502 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:02.453509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:02.453512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:02.453721 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:02.454165 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:02.454171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:02.454175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:02.454527 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:02.454679 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.455158 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.455384 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:44:02.455849 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.439672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:34335 (local address 127.8.218.195:35995)
0928 10:44:02.440495 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:02.440518 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:02.440691 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:02.441052 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:02.441057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:02.441731 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:02.442129 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.442144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.443211 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.443216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.444961 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.444979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.446005 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.446013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.446224 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.447329 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:02.447443 (+ 114us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:02.448977 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:02.452660 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:02.452667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:02.452674 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:02.453305 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:02.453873 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:02.453879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:02.453886 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:02.454047 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:02.454679 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:02.454685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:02.454897 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 10:44:02.455035 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 10:44:02.455396 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":444,"threads_started":1}
I20260928 10:44:02.465117 10545 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:33479
I20260928 10:44:02.465143 10680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:33479 every 8 connection(s)
I20260928 10:44:02.466531 10545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:02.469070 10681 heartbeater.cc:347] Connected to a master server at 127.8.218.254:34335
I20260928 10:44:02.469367 10681 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:02.470276 10681 heartbeater.cc:511] Master 127.8.218.254:34335 requested a full tablet report, sending...
I20260928 10:44:02.472818 10194 ts_manager.cc:194] Registered new tserver with Master: 66aaa86bb5e6468a96f4aec4a5fa219c (127.8.218.195:33479)
I20260928 10:44:02.474009 10194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:60049
I20260928 10:44:02.475327 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 10545
I20260928 10:44:02.475535 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:02.476063 9067 external_mini_cluster.cc:1428] Setting key 6afd36417f30f1b36101da616cf16c40
I20260928 10:44:02.479681 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:44:02 dist-test-slave-mn38 krb5kdc[10149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:44:02 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:02.528554 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.513657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.8.218.254:34335)
0928 10:44:02.513978 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.513989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.514246 (+ 257us) server_negotiation.cc:407] Connection header received
0928 10:44:02.514422 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.514428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.514504 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.514594 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:02.515968 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.516715 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.518264 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.518684 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.518799 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.522951 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:02.522956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:02.522960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:02.523007 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:02.525664 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:02.526489 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:02.526494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:02.526497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:02.526667 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:02.527146 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:02.527149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:02.527152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:02.527517 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:02.527704 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.528116 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.528296 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:44:02.532961 10194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:02.540638 10194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:02.568905 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.556636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.8.218.194:35617)
0928 10:44:02.560321 (+ 3685us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.560328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.560358 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:44:02.560479 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.560487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.560788 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.561010 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:02.561955 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.563451 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.565949 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.567602 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.567761 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.568022 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.568373 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.568584 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3540,"thread_start_us":3435,"threads_started":1}
I20260928 10:44:02.569235 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.556067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35617 (local address 127.0.0.1:40112)
0928 10:44:02.556901 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:02.556919 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:02.557027 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:02.561065 (+ 4038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:02.561087 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:02.561130 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:02.561681 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.561701 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.563603 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.563608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.565724 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.565739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.567848 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.567855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.568095 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.568150 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 10:44:02.568303 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 10:44:02.568514 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":1,"thread_start_us":5304,"threads_started":1}
I20260928 10:44:02.570000 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.556428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:35735 (local address 127.0.0.1:55878)
0928 10:44:02.561715 (+ 5287us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:02.561739 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:02.561895 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:02.562989 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:02.562994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:02.563023 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:02.563431 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.563442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.565136 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.565141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.567208 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.567223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.569271 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.569278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.569477 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.569509 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:44:02.569585 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 10:44:02.569724 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5133,"mutex_wait_us":50,"thread_start_us":668,"threads_started":1}
I20260928 10:44:02.570650 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.556939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.8.218.193:35735)
0928 10:44:02.559963 (+ 3024us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.559971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.562039 (+ 2068us) server_negotiation.cc:407] Connection header received
0928 10:44:02.562168 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.562176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.562522 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.562740 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:02.563613 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.564991 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.567387 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.568772 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.568925 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.569138 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.570226 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.570400 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2848,"thread_start_us":2679,"threads_started":1}
I20260928 10:44:02.571830 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.555277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33479 (local address 127.0.0.1:47726)
0928 10:44:02.556045 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:02.556082 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:02.556255 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:02.564329 (+ 8074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:02.564334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:02.564362 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:02.564738 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.564750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.566980 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.566986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.569047 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.569062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.570998 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.571005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.571197 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.571232 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:44:02.571322 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 10:44:02.571476 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":325,"threads_started":1}
I20260928 10:44:02.572693 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.555597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.8.218.195:33479)
0928 10:44:02.563630 (+ 8033us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.563639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.563674 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:44:02.563787 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.563795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.564086 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.564332 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:02.564946 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.566807 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.569243 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.570734 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.570917 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.571165 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.572146 (+ 981us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.572377 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7850,"thread_start_us":7658,"threads_started":1}
I20260928 10:44:02.574074 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 16556f80dc8e490896790101382b8739 (DEFAULT_TABLE table=test-table [id=5f4a347d4aba42ee96f1c1cde5c057c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:02.575640 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16556f80dc8e490896790101382b8739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:02.575229 10331 tablet_service.cc:1467] Processing CreateTablet for tablet 16556f80dc8e490896790101382b8739 (DEFAULT_TABLE table=test-table [id=5f4a347d4aba42ee96f1c1cde5c057c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:02.576944 10611 tablet_service.cc:1467] Processing CreateTablet for tablet 16556f80dc8e490896790101382b8739 (DEFAULT_TABLE table=test-table [id=5f4a347d4aba42ee96f1c1cde5c057c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:02.577978 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16556f80dc8e490896790101382b8739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:02.578588 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16556f80dc8e490896790101382b8739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:02.589183 10700 tablet_bootstrap.cc:492] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Bootstrap starting.
I20260928 10:44:02.593223 10700 tablet_bootstrap.cc:654] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:02.594158 10701 tablet_bootstrap.cc:492] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Bootstrap starting.
I20260928 10:44:02.594660 10700 log.cc:824] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:02.595115 10702 tablet_bootstrap.cc:492] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Bootstrap starting.
I20260928 10:44:02.597695 10700 tablet_bootstrap.cc:492] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: No bootstrap required, opened a new log
I20260928 10:44:02.598270 10700 ts_tablet_manager.cc:1397] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 10:44:02.599172 10702 tablet_bootstrap.cc:654] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:02.599535 10701 tablet_bootstrap.cc:654] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:02.600630 10702 log.cc:824] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:02.600668 10701 log.cc:824] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:02.604070 10701 tablet_bootstrap.cc:492] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: No bootstrap required, opened a new log
I20260928 10:44:02.604374 10701 ts_tablet_manager.cc:1397] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260928 10:44:02.606536 10702 tablet_bootstrap.cc:492] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: No bootstrap required, opened a new log
I20260928 10:44:02.606957 10702 ts_tablet_manager.cc:1397] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 10:44:02.607626 10700 raft_consensus.cc:348] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.608186 10700 raft_consensus.cc:374] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:02.608305 10700 raft_consensus.cc:729] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1908143790fe4a65acd0aa6fbc3fdb0d, State: Initialized, Role: FOLLOWER
I20260928 10:44:02.609117 10700 consensus_queue.cc:260] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.610818 10541 heartbeater.cc:503] Master 127.8.218.254:34335 was elected leader, sending a full tablet report...
I20260928 10:44:02.611740 10700 ts_tablet_manager.cc:1428] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260928 10:44:02.612877 10542 tablet.cc:2369] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:44:02.614570 10701 raft_consensus.cc:348] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.615132 10701 raft_consensus.cc:374] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:02.615257 10701 raft_consensus.cc:729] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab041a929744592828a1b43ab45802f, State: Initialized, Role: FOLLOWER
I20260928 10:44:02.616349 10701 consensus_queue.cc:260] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.618489 10701 ts_tablet_manager.cc:1428] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 10:44:02.619290 10702 raft_consensus.cc:348] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.619891 10702 raft_consensus.cc:374] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:02.620011 10702 raft_consensus.cc:729] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66aaa86bb5e6468a96f4aec4a5fa219c, State: Initialized, Role: FOLLOWER
I20260928 10:44:02.620783 10702 consensus_queue.cc:260] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.622092 10681 heartbeater.cc:503] Master 127.8.218.254:34335 was elected leader, sending a full tablet report...
I20260928 10:44:02.622941 10702 ts_tablet_manager.cc:1428] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
W20260928 10:44:02.718240 10682 tablet.cc:2369] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:44:02.777848 10402 tablet.cc:2369] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:44:02.890116 10707 raft_consensus.cc:482] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:02.890482 10707 raft_consensus.cc:504] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.892455 10707 leader_election.cc:288] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66aaa86bb5e6468a96f4aec4a5fa219c (127.8.218.195:33479), 1908143790fe4a65acd0aa6fbc3fdb0d (127.8.218.194:35617)
I20260928 10:44:02.902750 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.893230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:42799 (local address 127.8.218.194:35617)
0928 10:44:02.893530 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.893536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.893773 (+ 237us) server_negotiation.cc:407] Connection header received
0928 10:44:02.893917 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.893923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.894003 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.894105 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:02.894963 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.896154 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.899790 (+ 3636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.901089 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.901523 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.901612 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.902323 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.902484 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260928 10:44:02.902750 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.892835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:36819 (local address 127.8.218.195:33479)
0928 10:44:02.893041 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.893049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.893604 (+ 555us) server_negotiation.cc:407] Connection header received
0928 10:44:02.893936 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.893941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.894015 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.894104 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:02.894953 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.896154 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.899795 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.901089 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.901523 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.901612 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.902323 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.902506 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:44:02.902887 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.892796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35617 (local address 127.8.218.193:42799)
0928 10:44:02.893678 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:02.893693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:02.893793 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:02.894124 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:02.894130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:02.894224 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:02.894777 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.894787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.896354 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.896360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.899427 (+ 3067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.899459 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.901695 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.901705 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.902001 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.902056 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 10:44:02.902190 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 10:44:02.902393 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":243,"spinlock_wait_cycles":28544,"thread_start_us":556,"threads_started":1}
I20260928 10:44:02.903102 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.892705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33479 (local address 127.8.218.193:36819)
0928 10:44:02.893421 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:02.893489 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:02.893688 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:02.894124 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:02.894130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:02.894168 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:02.894730 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.894745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.896354 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.896360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.899427 (+ 3067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:02.899460 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.901704 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.901710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:02.901992 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.902053 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 10:44:02.902190 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 10:44:02.902393 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":258,"thread_start_us":326,"threads_started":1}
I20260928 10:44:02.904103 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16556f80dc8e490896790101382b8739" candidate_uuid: "7ab041a929744592828a1b43ab45802f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" is_pre_election: true
I20260928 10:44:02.904407 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16556f80dc8e490896790101382b8739" candidate_uuid: "7ab041a929744592828a1b43ab45802f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" is_pre_election: true
I20260928 10:44:02.904803 10491 raft_consensus.cc:2454] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ab041a929744592828a1b43ab45802f in term 0.
I20260928 10:44:02.904992 10631 raft_consensus.cc:2454] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ab041a929744592828a1b43ab45802f in term 0.
I20260928 10:44:02.905989 10282 leader_election.cc:302] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1908143790fe4a65acd0aa6fbc3fdb0d, 7ab041a929744592828a1b43ab45802f; no voters:
I20260928 10:44:02.906519 10707 raft_consensus.cc:2781] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:02.906706 10707 raft_consensus.cc:482] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:02.906836 10707 raft_consensus.cc:3037] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:02.909541 10707 raft_consensus.cc:504] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.910337 10707 leader_election.cc:288] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [CANDIDATE]: Term 1 election: Requested vote from peers 66aaa86bb5e6468a96f4aec4a5fa219c (127.8.218.195:33479), 1908143790fe4a65acd0aa6fbc3fdb0d (127.8.218.194:35617)
I20260928 10:44:02.910969 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16556f80dc8e490896790101382b8739" candidate_uuid: "7ab041a929744592828a1b43ab45802f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c"
I20260928 10:44:02.910984 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16556f80dc8e490896790101382b8739" candidate_uuid: "7ab041a929744592828a1b43ab45802f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d"
I20260928 10:44:02.911183 10631 raft_consensus.cc:3037] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:02.911331 10491 raft_consensus.cc:3037] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:02.913827 10631 raft_consensus.cc:2454] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ab041a929744592828a1b43ab45802f in term 1.
I20260928 10:44:02.913915 10491 raft_consensus.cc:2454] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ab041a929744592828a1b43ab45802f in term 1.
I20260928 10:44:02.914466 10283 leader_election.cc:302] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1908143790fe4a65acd0aa6fbc3fdb0d, 66aaa86bb5e6468a96f4aec4a5fa219c, 7ab041a929744592828a1b43ab45802f; no voters:
I20260928 10:44:02.914927 10707 raft_consensus.cc:2781] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:02.915462 10707 raft_consensus.cc:686] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 1 LEADER]: Becoming Leader. State: Replica: 7ab041a929744592828a1b43ab45802f, State: Running, Role: LEADER
I20260928 10:44:02.915980 10707 consensus_queue.cc:237] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:02.921025 10194 catalog_manager.cc:5495] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ab041a929744592828a1b43ab45802f (127.8.218.193). New cstate: current_term: 1 leader_uuid: "7ab041a929744592828a1b43ab45802f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } health_report { overall_health: UNKNOWN } } }
I20260928 10:44:02.947788 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:02.941859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.8.218.193:35735)
0928 10:44:02.942070 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:02.942074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:02.942095 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:44:02.942415 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:02.942419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:02.942526 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:02.942647 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:02.942952 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.943673 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.945139 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:02.945633 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:02.945750 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:02.946406 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:02.947131 (+ 725us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:02.947250 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:02.947412 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 10:44:02.947507 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:44:02.957096 10631 raft_consensus.cc:1261] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Refusing update from remote peer 7ab041a929744592828a1b43ab45802f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:02.957096 10491 raft_consensus.cc:1261] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Refusing update from remote peer 7ab041a929744592828a1b43ab45802f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:02.958360 10707 consensus_queue.cc:1035] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:02.958680 10711 consensus_queue.cc:1035] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:02.994087 10194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:02.994371 10194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:02.999336 10194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a: Sending DeleteTablet for 3 replicas of tablet 16556f80dc8e490896790101382b8739
I20260928 10:44:03.000644 10611 tablet_service.cc:1514] Processing DeleteTablet for tablet 16556f80dc8e490896790101382b8739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:02 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:47726
I20260928 10:44:03.000664 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 16556f80dc8e490896790101382b8739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:02 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:40112
I20260928 10:44:03.000700 10331 tablet_service.cc:1514] Processing DeleteTablet for tablet 16556f80dc8e490896790101382b8739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:02 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:55878
I20260928 10:44:03.005179 10194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:03.006151 10194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:03.013054 10726 tablet_replica.cc:330] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: stopping tablet replica
I20260928 10:44:03.013635 10726 raft_consensus.cc:2229] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:03.014523 10726 raft_consensus.cc:2258] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:03.019613 10726 ts_tablet_manager.cc:1905] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:03.021523 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 8b198b8584c34544bd724fbad4b3d5e8 (DEFAULT_TABLE table=test-table [id=f4c95e1057f24c37a6b01aba47f07ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:03.022123 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b198b8584c34544bd724fbad4b3d5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:03.021991 10610 tablet_service.cc:1467] Processing CreateTablet for tablet 8b198b8584c34544bd724fbad4b3d5e8 (DEFAULT_TABLE table=test-table [id=f4c95e1057f24c37a6b01aba47f07ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:03.022498 10610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b198b8584c34544bd724fbad4b3d5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:03.029032 10700 tablet_bootstrap.cc:492] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: Bootstrap starting.
I20260928 10:44:03.029072 10702 tablet_bootstrap.cc:492] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: Bootstrap starting.
I20260928 10:44:03.030414 10331 tablet_service.cc:1467] Processing CreateTablet for tablet 8b198b8584c34544bd724fbad4b3d5e8 (DEFAULT_TABLE table=test-table [id=f4c95e1057f24c37a6b01aba47f07ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:03.030716 10700 tablet_bootstrap.cc:654] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:03.030829 10702 tablet_bootstrap.cc:654] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:03.030928 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b198b8584c34544bd724fbad4b3d5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:03.033123 10700 tablet_bootstrap.cc:492] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: No bootstrap required, opened a new log
I20260928 10:44:03.033289 10700 ts_tablet_manager.cc:1397] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 10:44:03.034129 10700 raft_consensus.cc:348] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.034360 10700 raft_consensus.cc:374] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:03.034443 10700 raft_consensus.cc:729] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1908143790fe4a65acd0aa6fbc3fdb0d, State: Initialized, Role: FOLLOWER
I20260928 10:44:03.035200 10700 consensus_queue.cc:260] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.036904 10700 ts_tablet_manager.cc:1428] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 10:44:03.042693 10702 tablet_bootstrap.cc:492] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: No bootstrap required, opened a new log
I20260928 10:44:03.042894 10702 ts_tablet_manager.cc:1397] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260928 10:44:03.043857 10702 raft_consensus.cc:348] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.044107 10702 raft_consensus.cc:374] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:03.044189 10702 raft_consensus.cc:729] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66aaa86bb5e6468a96f4aec4a5fa219c, State: Initialized, Role: FOLLOWER
I20260928 10:44:03.045085 10724 tablet_replica.cc:330] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: stopping tablet replica
I20260928 10:44:03.045230 10725 tablet_replica.cc:330] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: stopping tablet replica
I20260928 10:44:03.045645 10701 tablet_bootstrap.cc:492] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: Bootstrap starting.
I20260928 10:44:03.045877 10725 raft_consensus.cc:2229] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:03.046504 10725 raft_consensus.cc:2258] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:03.047351 10724 raft_consensus.cc:2229] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:03.047561 10701 tablet_bootstrap.cc:654] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:03.047981 10724 raft_consensus.cc:2258] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:03.050437 10725 ts_tablet_manager.cc:1905] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:03.050820 10702 consensus_queue.cc:260] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.051638 10702 ts_tablet_manager.cc:1428] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20260928 10:44:03.055977 10724 ts_tablet_manager.cc:1905] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:03.059351 10725 ts_tablet_manager.cc:1918] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:03.059633 10725 log.cc:1196] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/16556f80dc8e490896790101382b8739
I20260928 10:44:03.060236 10725 ts_tablet_manager.cc:1939] T 16556f80dc8e490896790101382b8739 P 1908143790fe4a65acd0aa6fbc3fdb0d: Deleting consensus metadata
I20260928 10:44:03.062429 10179 catalog_manager.cc:4841] TS 1908143790fe4a65acd0aa6fbc3fdb0d (127.8.218.194:35617): tablet 16556f80dc8e490896790101382b8739 (table test-table [id=5f4a347d4aba42ee96f1c1cde5c057c8]) successfully deleted
I20260928 10:44:03.064004 10724 ts_tablet_manager.cc:1918] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:03.064193 10724 log.cc:1196] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/16556f80dc8e490896790101382b8739
I20260928 10:44:03.064590 10701 tablet_bootstrap.cc:492] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: No bootstrap required, opened a new log
I20260928 10:44:03.064715 10724 ts_tablet_manager.cc:1939] T 16556f80dc8e490896790101382b8739 P 66aaa86bb5e6468a96f4aec4a5fa219c: Deleting consensus metadata
I20260928 10:44:03.064751 10701 ts_tablet_manager.cc:1397] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20260928 10:44:03.065593 10701 raft_consensus.cc:348] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.065788 10701 raft_consensus.cc:374] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:03.065891 10701 raft_consensus.cc:729] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab041a929744592828a1b43ab45802f, State: Initialized, Role: FOLLOWER
I20260928 10:44:03.066201 10701 consensus_queue.cc:260] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.066779 10180 catalog_manager.cc:4841] TS 66aaa86bb5e6468a96f4aec4a5fa219c (127.8.218.195:33479): tablet 16556f80dc8e490896790101382b8739 (table test-table [id=5f4a347d4aba42ee96f1c1cde5c057c8]) successfully deleted
I20260928 10:44:03.067133 10701 ts_tablet_manager.cc:1428] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 10:44:03.069043 10726 ts_tablet_manager.cc:1918] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:03.069283 10726 log.cc:1196] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/16556f80dc8e490896790101382b8739
I20260928 10:44:03.069914 10726 ts_tablet_manager.cc:1939] T 16556f80dc8e490896790101382b8739 P 7ab041a929744592828a1b43ab45802f: Deleting consensus metadata
I20260928 10:44:03.071918 10180 catalog_manager.cc:4841] TS 7ab041a929744592828a1b43ab45802f (127.8.218.193:35735): tablet 16556f80dc8e490896790101382b8739 (table test-table [id=5f4a347d4aba42ee96f1c1cde5c057c8]) successfully deleted
I20260928 10:44:03.098758 10708 raft_consensus.cc:482] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:03.099048 10708 raft_consensus.cc:504] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.101189 10708 leader_election.cc:288] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ab041a929744592828a1b43ab45802f (127.8.218.193:35735), 1908143790fe4a65acd0aa6fbc3fdb0d (127.8.218.194:35617)
I20260928 10:44:03.110388 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.101806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:56907 (local address 127.8.218.193:35735)
0928 10:44:03.102054 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.102061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.102703 (+ 642us) server_negotiation.cc:407] Connection header received
0928 10:44:03.103030 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.103036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.103125 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.103198 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:03.104003 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.105005 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.108181 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.109054 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.109257 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.109350 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.109843 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.110016 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:44:03.110574 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.101537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35617 (local address 127.8.218.195:43955)
0928 10:44:03.103125 (+ 1588us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:03.103141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:03.103260 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:03.103617 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:03.103621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:03.103643 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:03.104038 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.104051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.105541 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.105547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.107870 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.107895 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.109732 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.109740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.109949 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.109983 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:44:03.110078 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 10:44:03.110230 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"mutex_wait_us":61,"thread_start_us":740,"threads_started":1}
I20260928 10:44:03.111073 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.101675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:43955 (local address 127.8.218.194:35617)
0928 10:44:03.101882 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.101889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.103241 (+ 1352us) server_negotiation.cc:407] Connection header received
0928 10:44:03.103374 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.103380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.103466 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.103578 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:03.104219 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.105391 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.108123 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.109425 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.109710 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.109789 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.110138 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.110297 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:44:03.111251 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.101382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:35735 (local address 127.8.218.195:56907)
0928 10:44:03.102586 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:03.102606 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:03.102758 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:03.103402 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:03.103405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:03.103427 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:03.103774 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.103789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.105278 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.105282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.107870 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.107895 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.109301 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.109309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.109556 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.109659 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 10:44:03.110784 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0928 10:44:03.110939 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":87,"thread_start_us":872,"threads_started":1}
I20260928 10:44:03.111790 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b198b8584c34544bd724fbad4b3d5e8" candidate_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" is_pre_election: true
I20260928 10:44:03.112115 10491 raft_consensus.cc:2454] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66aaa86bb5e6468a96f4aec4a5fa219c in term 0.
I20260928 10:44:03.112573 10351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b198b8584c34544bd724fbad4b3d5e8" candidate_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab041a929744592828a1b43ab45802f" is_pre_election: true
I20260928 10:44:03.112910 10562 leader_election.cc:302] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1908143790fe4a65acd0aa6fbc3fdb0d, 66aaa86bb5e6468a96f4aec4a5fa219c; no voters:
I20260928 10:44:03.113027 10351 raft_consensus.cc:2454] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66aaa86bb5e6468a96f4aec4a5fa219c in term 0.
I20260928 10:44:03.113480 10708 raft_consensus.cc:2781] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:03.113610 10708 raft_consensus.cc:482] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:03.113705 10708 raft_consensus.cc:3037] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:03.116094 10708 raft_consensus.cc:504] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.116784 10708 leader_election.cc:288] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [CANDIDATE]: Term 1 election: Requested vote from peers 7ab041a929744592828a1b43ab45802f (127.8.218.193:35735), 1908143790fe4a65acd0aa6fbc3fdb0d (127.8.218.194:35617)
I20260928 10:44:03.117566 10351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b198b8584c34544bd724fbad4b3d5e8" candidate_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab041a929744592828a1b43ab45802f"
I20260928 10:44:03.117640 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b198b8584c34544bd724fbad4b3d5e8" candidate_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d"
I20260928 10:44:03.117841 10351 raft_consensus.cc:3037] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:03.117846 10491 raft_consensus.cc:3037] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:03.120021 10491 raft_consensus.cc:2454] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66aaa86bb5e6468a96f4aec4a5fa219c in term 1.
I20260928 10:44:03.120563 10562 leader_election.cc:302] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1908143790fe4a65acd0aa6fbc3fdb0d, 66aaa86bb5e6468a96f4aec4a5fa219c; no voters:
I20260928 10:44:03.120870 10708 raft_consensus.cc:2781] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:03.121306 10351 raft_consensus.cc:2454] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66aaa86bb5e6468a96f4aec4a5fa219c in term 1.
I20260928 10:44:03.121376 10708 raft_consensus.cc:686] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 LEADER]: Becoming Leader. State: Replica: 66aaa86bb5e6468a96f4aec4a5fa219c, State: Running, Role: LEADER
I20260928 10:44:03.121856 10708 consensus_queue.cc:237] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } }
I20260928 10:44:03.125800 10194 catalog_manager.cc:5495] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c reported cstate change: term changed from 0 to 1, leader changed from <none> to 66aaa86bb5e6468a96f4aec4a5fa219c (127.8.218.195). New cstate: current_term: 1 leader_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } health_report { overall_health: HEALTHY } } }
Sep 28 10:44:03 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592240, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:03.166925 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.152167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:34335 (local address 127.0.0.1:35488)
0928 10:44:03.153204 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:03.153220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:03.153408 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:03.153783 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:03.153788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:03.154731 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:03.154985 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.154998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.156175 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.156179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.157207 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.157221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.158081 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.158087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.158252 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.158933 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:03.158952 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:03.161155 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:03.164177 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:03.164193 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:03.164202 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:03.164651 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:03.165010 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:03.165015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:03.165019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:03.165153 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:03.165718 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:03.165727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:03.165919 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 10:44:03.166093 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 10:44:03.166427 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":728,"threads_started":1}
I20260928 10:44:03.166944 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.152171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.8.218.254:34335)
0928 10:44:03.153064 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.153075 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.153335 (+ 260us) server_negotiation.cc:407] Connection header received
0928 10:44:03.153557 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.153563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.153626 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.153733 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:03.155182 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.155994 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.157364 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.157909 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.158030 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.161366 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:03.161371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:03.161374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:03.161405 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:03.164028 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.164769 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.164781 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.164784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.164904 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.165253 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.165256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.165258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.165595 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:03.165813 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.166185 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.166524 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":693,"thread_start_us":415,"threads_started":1}
I20260928 10:44:03.172853 10194 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:35488:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:44:03.173955 10194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:03.181340 10331 tablet_service.cc:1467] Processing CreateTablet for tablet d9fbd41876454b7a9688921db02f6a98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e235e1103d64996b66d21922a0e091d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:44:03.181625 10611 tablet_service.cc:1467] Processing CreateTablet for tablet d9fbd41876454b7a9688921db02f6a98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e235e1103d64996b66d21922a0e091d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:44:03.182188 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fbd41876454b7a9688921db02f6a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:03.182224 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fbd41876454b7a9688921db02f6a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:03.181996 10471 tablet_service.cc:1467] Processing CreateTablet for tablet d9fbd41876454b7a9688921db02f6a98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e235e1103d64996b66d21922a0e091d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:44:03.182577 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fbd41876454b7a9688921db02f6a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:03.190402 10701 tablet_bootstrap.cc:492] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: Bootstrap starting.
I20260928 10:44:03.190430 10702 tablet_bootstrap.cc:492] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: Bootstrap starting.
I20260928 10:44:03.191937 10701 tablet_bootstrap.cc:654] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:03.192198 10702 tablet_bootstrap.cc:654] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:03.194489 10701 tablet_bootstrap.cc:492] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: No bootstrap required, opened a new log
I20260928 10:44:03.194706 10701 ts_tablet_manager.cc:1397] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 10:44:03.194926 10702 tablet_bootstrap.cc:492] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: No bootstrap required, opened a new log
I20260928 10:44:03.195106 10702 ts_tablet_manager.cc:1397] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260928 10:44:03.195458 10701 raft_consensus.cc:348] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.195731 10701 raft_consensus.cc:374] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:03.195822 10701 raft_consensus.cc:729] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab041a929744592828a1b43ab45802f, State: Initialized, Role: FOLLOWER
I20260928 10:44:03.195869 10702 raft_consensus.cc:348] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.196137 10702 raft_consensus.cc:374] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:03.196215 10702 raft_consensus.cc:729] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66aaa86bb5e6468a96f4aec4a5fa219c, State: Initialized, Role: FOLLOWER
I20260928 10:44:03.196110 10701 consensus_queue.cc:260] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.196799 10701 ts_tablet_manager.cc:1428] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 10:44:03.196622 10707 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.196945 10707 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:03.197196 10702 consensus_queue.cc:260] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.197790 10708 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.198136 10708 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:03.199733 10702 ts_tablet_manager.cc:1428] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 10:44:03.203912 10700 tablet_bootstrap.cc:492] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: Bootstrap starting.
I20260928 10:44:03.205896 10700 tablet_bootstrap.cc:654] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:03.208696 10700 tablet_bootstrap.cc:492] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: No bootstrap required, opened a new log
I20260928 10:44:03.208904 10700 ts_tablet_manager.cc:1397] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 10:44:03.209821 10700 raft_consensus.cc:348] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.210141 10700 raft_consensus.cc:374] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:03.210224 10700 raft_consensus.cc:729] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1908143790fe4a65acd0aa6fbc3fdb0d, State: Initialized, Role: FOLLOWER
I20260928 10:44:03.210587 10700 consensus_queue.cc:260] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.211414 10706 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.211864 10706 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:03.212309 10700 ts_tablet_manager.cc:1428] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 10:44:03.341096 10706 raft_consensus.cc:482] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:03.341300 10706 raft_consensus.cc:504] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.342904 10706 leader_election.cc:288] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ab041a929744592828a1b43ab45802f (127.8.218.193:35735), 66aaa86bb5e6468a96f4aec4a5fa219c (127.8.218.195:33479)
I20260928 10:44:03.351168 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.343315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:38653 (local address 127.8.218.193:35735)
0928 10:44:03.343510 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.343517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.343836 (+ 319us) server_negotiation.cc:407] Connection header received
0928 10:44:03.344062 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.344067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.344141 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.344210 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:03.345065 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.346104 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.348641 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.350094 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.350300 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.350389 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.350736 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.350893 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:44:03.351184 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.343215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:35735 (local address 127.8.218.194:38653)
0928 10:44:03.343733 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:03.343748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:03.343892 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:03.344461 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:03.344465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:03.344498 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:03.344886 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.344896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.346236 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.346240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.348405 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.348423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.350344 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.350351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.350527 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.350562 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:44:03.350700 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 10:44:03.350853 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":262,"threads_started":1}
I20260928 10:44:03.351526 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.344093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:33539 (local address 127.8.218.195:33479)
0928 10:44:03.344263 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.344269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.344863 (+ 594us) server_negotiation.cc:407] Connection header received
0928 10:44:03.345052 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.345059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.345142 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.345240 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:03.345841 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.347031 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.349430 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.350408 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.350561 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.350625 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.351099 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.351243 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:44:03.352065 10351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fbd41876454b7a9688921db02f6a98" candidate_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab041a929744592828a1b43ab45802f" is_pre_election: true
I20260928 10:44:03.352335 10351 raft_consensus.cc:2454] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1908143790fe4a65acd0aa6fbc3fdb0d in term 0.
I20260928 10:44:03.352371 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.343927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33479 (local address 127.8.218.194:33539)
0928 10:44:03.344723 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:03.344748 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:03.344890 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:03.345266 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:03.345271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:03.345296 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:03.345653 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.345664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.347167 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.347173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.349260 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.349286 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.350640 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.350647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.350899 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.350933 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:44:03.351818 (+ 885us) client_negotiation.cc:240] Negotiation successful
0928 10:44:03.352041 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":476,"threads_started":1}
I20260928 10:44:03.352841 10423 leader_election.cc:302] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1908143790fe4a65acd0aa6fbc3fdb0d, 7ab041a929744592828a1b43ab45802f; no voters:
I20260928 10:44:03.352980 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fbd41876454b7a9688921db02f6a98" candidate_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" is_pre_election: true
I20260928 10:44:03.353207 10631 raft_consensus.cc:2454] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1908143790fe4a65acd0aa6fbc3fdb0d in term 0.
I20260928 10:44:03.353333 10706 raft_consensus.cc:2781] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:03.353482 10706 raft_consensus.cc:482] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:03.353576 10706 raft_consensus.cc:3037] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:03.355845 10706 raft_consensus.cc:504] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.356381 10706 leader_election.cc:288] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [CANDIDATE]: Term 1 election: Requested vote from peers 7ab041a929744592828a1b43ab45802f (127.8.218.193:35735), 66aaa86bb5e6468a96f4aec4a5fa219c (127.8.218.195:33479)
I20260928 10:44:03.356964 10351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fbd41876454b7a9688921db02f6a98" candidate_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab041a929744592828a1b43ab45802f"
I20260928 10:44:03.357051 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fbd41876454b7a9688921db02f6a98" candidate_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c"
I20260928 10:44:03.357213 10351 raft_consensus.cc:3037] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:03.357254 10631 raft_consensus.cc:3037] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:03.360311 10351 raft_consensus.cc:2454] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1908143790fe4a65acd0aa6fbc3fdb0d in term 1.
I20260928 10:44:03.360311 10631 raft_consensus.cc:2454] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1908143790fe4a65acd0aa6fbc3fdb0d in term 1.
I20260928 10:44:03.360811 10423 leader_election.cc:302] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1908143790fe4a65acd0aa6fbc3fdb0d, 66aaa86bb5e6468a96f4aec4a5fa219c; no voters:
I20260928 10:44:03.361141 10706 raft_consensus.cc:2781] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:03.361668 10706 raft_consensus.cc:686] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 LEADER]: Becoming Leader. State: Replica: 1908143790fe4a65acd0aa6fbc3fdb0d, State: Running, Role: LEADER
I20260928 10:44:03.362195 10706 consensus_queue.cc:237] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } }
I20260928 10:44:03.364487 10746 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: TxnStatusTablet state changed. Reason: New leader 1908143790fe4a65acd0aa6fbc3fdb0d. Latest consensus state: current_term: 1 leader_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.364996 10746 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:44:03.367116 10194 catalog_manager.cc:5495] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1908143790fe4a65acd0aa6fbc3fdb0d (127.8.218.194). New cstate: current_term: 1 leader_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } health_report { overall_health: HEALTHY } } }
I20260928 10:44:03.372123 10748 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:44:03.372444 10748 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:44:03.419339 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.413031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35617 (local address 127.0.0.1:40126)
0928 10:44:03.413317 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:03.413342 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:03.413594 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:03.414191 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:03.414196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:03.414209 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:44:03.414397 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.414406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.415545 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.415550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.416657 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.416673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.417528 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.417535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.417697 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.417773 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:44:03.418742 (+ 969us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:44:03.418766 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 10:44:03.418849 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 10:44:03.419031 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 10:44:03.419333 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.413520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.8.218.194:35617)
0928 10:44:03.413755 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.413761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.413798 (+ 37us) server_negotiation.cc:407] Connection header received
0928 10:44:03.413912 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.413915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.414020 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.414146 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:03.414570 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.415395 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.416842 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.417314 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.417465 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.417988 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:03.418624 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:03.418774 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.418888 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.418986 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 10:44:03.423941 10631 raft_consensus.cc:1261] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Refusing update from remote peer 1908143790fe4a65acd0aa6fbc3fdb0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:03.424721 10706 consensus_queue.cc:1035] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:03.424924 10351 raft_consensus.cc:1261] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Refusing update from remote peer 1908143790fe4a65acd0aa6fbc3fdb0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:03.425923 10706 consensus_queue.cc:1035] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:03.430604 10708 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: TxnStatusTablet state changed. Reason: New leader 1908143790fe4a65acd0aa6fbc3fdb0d. Latest consensus state: current_term: 1 leader_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.430946 10708 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:03.431875 10716 mvcc.cc:205] Tried to move back new op lower bound from 7334265829056315392 to 7334265828816318464. Current Snapshot: MvccSnapshot[applied={T|T < 7334265829056315392}]
I20260928 10:44:03.433185 10706 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.433548 10706 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:44:03.433974 10715 mvcc.cc:205] Tried to move back new op lower bound from 7334265829056315392 to 7334265828816318464. Current Snapshot: MvccSnapshot[applied={T|T < 7334265829056315392 or (T in {7334265829056315392})}]
I20260928 10:44:03.435870 10708 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.436151 10708 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 66aaa86bb5e6468a96f4aec4a5fa219c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:03.449667 10707 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: TxnStatusTablet state changed. Reason: New leader 1908143790fe4a65acd0aa6fbc3fdb0d. Latest consensus state: current_term: 1 leader_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.450006 10707 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:03.452641 10706 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.453095 10706 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 1908143790fe4a65acd0aa6fbc3fdb0d: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:44:03.456422 10707 tablet_replica.cc:439] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 } } peers { permanent_uuid: "66aaa86bb5e6468a96f4aec4a5fa219c" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33479 } } peers { permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 } } }
I20260928 10:44:03.456703 10707 tablet_replica.cc:442] T d9fbd41876454b7a9688921db02f6a98 P 7ab041a929744592828a1b43ab45802f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:03.472132 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.450274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.8.218.195:33479)
0928 10:44:03.450506 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.450513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.462641 (+ 12128us) server_negotiation.cc:407] Connection header received
0928 10:44:03.462865 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.462869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.462954 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.463111 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:03.463912 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.465134 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.468209 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.468983 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.469094 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.469927 (+ 833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:03.470738 (+ 811us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:03.470874 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.471657 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.471800 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:44:03.516207 10708 consensus_queue.cc:1035] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1908143790fe4a65acd0aa6fbc3fdb0d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:03.522089 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.506001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:50533 (local address 127.8.218.194:35617)
0928 10:44:03.506217 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.506225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.514174 (+ 7949us) server_negotiation.cc:407] Connection header received
0928 10:44:03.514604 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.514610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.514708 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.514798 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:03.515626 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.516690 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.518895 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.520113 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.520242 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.521069 (+ 827us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:03.521256 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:03.521320 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.521580 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.521700 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:44:03.522927 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.505471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35617 (local address 127.8.218.195:50533)
0928 10:44:03.514028 (+ 8557us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:03.514049 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:03.514404 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:03.514998 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:03.515003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:03.515016 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:44:03.515347 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.515360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.516967 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.516972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.518589 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.518609 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.520484 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.520491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.520695 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.520801 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:44:03.521408 (+ 607us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:44:03.521420 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:44:03.522309 (+ 889us) client_negotiation.cc:240] Negotiation successful
0928 10:44:03.522472 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8366,"thread_start_us":8183,"threads_started":1}
I20260928 10:44:03.546295 10351 raft_consensus.cc:1261] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Refusing update from remote peer 66aaa86bb5e6468a96f4aec4a5fa219c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:03.547067 10708 consensus_queue.cc:1035] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab041a929744592828a1b43ab45802f" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:03.624006 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.608402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33479 (local address 127.8.218.194:33893)
0928 10:44:03.615114 (+ 6712us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:03.615142 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:03.615900 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:03.616632 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:03.616637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:03.616666 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:44:03.617038 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.617051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.618719 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.618729 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.620628 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:03.620654 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.621993 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.622001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:03.622207 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.622300 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:44:03.622998 (+ 698us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:44:03.623012 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 10:44:03.623097 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 10:44:03.623290 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6447,"thread_start_us":6901,"threads_started":1}
I20260928 10:44:03.624105 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.608030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:33893 (local address 127.8.218.195:33479)
0928 10:44:03.608208 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.608216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.615253 (+ 7037us) server_negotiation.cc:407] Connection header received
0928 10:44:03.616132 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.616139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.616325 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.616409 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:03.617241 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.618507 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.620862 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.621638 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.621753 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.622529 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:03.622705 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:03.623005 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.623707 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.623811 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:44:03.670814 10194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:03.671057 10194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:03.674196 10194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cb8343ac0ac4048a7829e0760364f7a: Sending DeleteTablet for 3 replicas of tablet 8b198b8584c34544bd724fbad4b3d5e8
I20260928 10:44:03.675246 10331 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b198b8584c34544bd724fbad4b3d5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:03 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:55878
I20260928 10:44:03.675733 10611 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b198b8584c34544bd724fbad4b3d5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:03 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:47726
I20260928 10:44:03.675933 10766 tablet_replica.cc:330] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: stopping tablet replica
I20260928 10:44:03.676141 10766 raft_consensus.cc:2229] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:03.676503 10766 raft_consensus.cc:2258] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:03.677373 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b198b8584c34544bd724fbad4b3d5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:03 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:40112
I20260928 10:44:03.691144 10766 ts_tablet_manager.cc:1905] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:03.699716 10766 ts_tablet_manager.cc:1918] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:03.699982 10767 tablet_replica.cc:330] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: stopping tablet replica
I20260928 10:44:03.700258 10767 raft_consensus.cc:2229] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:03.700915 10767 raft_consensus.cc:2258] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:03.702035 10766 log.cc:1196] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/8b198b8584c34544bd724fbad4b3d5e8
I20260928 10:44:03.702805 10766 ts_tablet_manager.cc:1939] T 8b198b8584c34544bd724fbad4b3d5e8 P 7ab041a929744592828a1b43ab45802f: Deleting consensus metadata
I20260928 10:44:03.704908 10180 catalog_manager.cc:4841] TS 7ab041a929744592828a1b43ab45802f (127.8.218.193:35735): tablet 8b198b8584c34544bd724fbad4b3d5e8 (table test-table [id=f4c95e1057f24c37a6b01aba47f07ac5]) successfully deleted
I20260928 10:44:03.706107 10767 ts_tablet_manager.cc:1905] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:03.709765 10768 tablet_replica.cc:330] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: stopping tablet replica
I20260928 10:44:03.710024 10768 raft_consensus.cc:2229] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:03.710330 10768 raft_consensus.cc:2258] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:03.712832 10768 ts_tablet_manager.cc:1905] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:03.719647 10768 ts_tablet_manager.cc:1918] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:03.719985 10768 log.cc:1196] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/8b198b8584c34544bd724fbad4b3d5e8
I20260928 10:44:03.720513 10768 ts_tablet_manager.cc:1939] T 8b198b8584c34544bd724fbad4b3d5e8 P 1908143790fe4a65acd0aa6fbc3fdb0d: Deleting consensus metadata
I20260928 10:44:03.721966 10179 catalog_manager.cc:4841] TS 1908143790fe4a65acd0aa6fbc3fdb0d (127.8.218.194:35617): tablet 8b198b8584c34544bd724fbad4b3d5e8 (table test-table [id=f4c95e1057f24c37a6b01aba47f07ac5]) successfully deleted
I20260928 10:44:03.726331 10767 ts_tablet_manager.cc:1918] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:03.726613 10767 log.cc:1196] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/8b198b8584c34544bd724fbad4b3d5e8
I20260928 10:44:03.727136 10767 ts_tablet_manager.cc:1939] T 8b198b8584c34544bd724fbad4b3d5e8 P 66aaa86bb5e6468a96f4aec4a5fa219c: Deleting consensus metadata
I20260928 10:44:03.728566 10180 catalog_manager.cc:4841] TS 66aaa86bb5e6468a96f4aec4a5fa219c (127.8.218.195:33479): tablet 8b198b8584c34544bd724fbad4b3d5e8 (table test-table [id=f4c95e1057f24c37a6b01aba47f07ac5]) successfully deleted
Sep 28 10:44:03 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:44:03.736618 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.698507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.8.218.193:35735)
0928 10:44:03.698693 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.698699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.701840 (+ 3141us) server_negotiation.cc:407] Connection header received
0928 10:44:03.702155 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.702161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.702271 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.702356 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:03.711008 (+ 8652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.711815 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.719697 (+ 7882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.720372 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.720503 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.729963 (+ 9460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:03.729981 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:03.729998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:03.730039 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:03.733252 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.734261 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.734318 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.734321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.734492 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.734956 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.734963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.734966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.735435 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:03.735790 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.736031 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.736281 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 10:44:03.737421 10396 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60962
I20260928 10:44:03.772352 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.740877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.8.218.254:34335)
0928 10:44:03.756984 (+ 16107us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.756992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.757603 (+ 611us) server_negotiation.cc:407] Connection header received
0928 10:44:03.757927 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.757934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.758015 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.758117 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:03.759962 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.761356 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.763113 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.763575 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.763702 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.766305 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:03.766310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:03.766314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:03.766371 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:03.769108 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.770080 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.770085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.770087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.770239 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.770648 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.770652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.770655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.770963 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:03.771212 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.771591 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.771869 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15913,"thread_start_us":16694,"threads_started":1}
W20260928 10:44:03.773041 10194 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37196
I20260928 10:44:03.776551 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10265
I20260928 10:44:03.793169 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.777892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.8.218.193:35735)
0928 10:44:03.778071 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.778078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.780683 (+ 2605us) server_negotiation.cc:407] Connection header received
0928 10:44:03.780881 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.780887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.780972 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.781093 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:03.782428 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.783160 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.784422 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.784838 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.784970 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.787377 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:03.787383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:03.787386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:03.787433 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:03.790221 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.791048 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.791055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.791059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.791212 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.791635 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.791641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.791644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.791959 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:03.792219 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.792553 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.792799 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 10:44:03.793882 10396 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60976
W20260928 10:44:03.794476 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10265: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:03.794664 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 10265
I20260928 10:44:03.815379 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10405
Sep 28 10:44:03 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:44:03.831746 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.816316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.8.218.194:35617)
0928 10:44:03.816535 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.816541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.816562 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:44:03.816758 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.816762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.816830 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.816929 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:03.818206 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.819010 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.820348 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.820838 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.820959 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.825179 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:03.825196 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:03.825213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:03.825272 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:03.828376 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.829207 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.829216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.829219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.829396 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.829981 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.829985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.829987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.830396 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:03.830706 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.831018 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.831278 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260928 10:44:03.832517 10536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44468
W20260928 10:44:03.833283 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10405: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:03.833484 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 10405
I20260928 10:44:03.858505 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10545
Sep 28 10:44:03 dist-test-slave-mn38 krb5kdc[10149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:44:03.874740 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:03.859430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.8.218.195:33479)
0928 10:44:03.859626 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:03.859631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:03.859652 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:44:03.859836 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:03.859840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:03.859904 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:03.859987 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:03.861250 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.862019 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.863260 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:03.863682 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:03.863802 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:03.868384 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:03.868399 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:03.868416 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:03.868460 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:03.871631 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.872462 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.872469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.872472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.872624 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:03.873023 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:03.873027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:03.873029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:03.873469 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:03.873802 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:03.874105 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 10:44:03.874365 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 10:44:03.875586 10676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33960
W20260928 10:44:03.876451 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:03.876631 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 10545
I20260928 10:44:03.892364 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10162
I20260928 10:44:03.893685 10234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:04.010365 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 10162
2026-09-28T10:44:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5200 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:04 dist-test-slave-mn38 krb5kdc[10778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:04 dist-test-slave-mn38 krb5kdc[10778](info): set up 2 sockets
Sep 28 10:44:04 dist-test-slave-mn38 krb5kdc[10778](info): commencing operation
krb5kdc: starting...
W20260928 10:44:05.050974 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/test-user.keytab.
W20260928 10:44:06.000794 9067 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.818s user 0.000s sys 0.004s
Sep 28 10:44:06 dist-test-slave-mn38 krb5kdc[10778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
W20260928 10:44:07.310072 9067 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.218.254: real 1.259s user 0.000s sys 0.009s
I20260928 10:44:07.311154 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:35751
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:35751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:07.675436 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:07.675809 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:07.675913 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:07.684772 10791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:07.684939 10791 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:44:07.684983 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:07.685038 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:07.685089 10791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:07.685153 10791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:07.696492 10791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:35751
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:35751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:07.697973 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:07.699519 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:07.712828 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:07.712852 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:07.712893 10799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:07.713264 10791 server_base.cc:1034] running on GCE node
I20260928 10:44:07.714493 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:07.716321 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:07.717666 10791 hybrid_clock.cc:648] HybridClock initialized: now 1790592247717580 us; error 62 us; skew 500 ppm
I20260928 10:44:07.718098 10791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:07 dist-test-slave-mn38 krb5kdc[10778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592247, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:07.730147 10791 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:44:07.731968 10791 webserver.cc:460] Webserver started at http://127.8.218.254:41637/ using document root <none> and password file <none>
I20260928 10:44:07.732736 10791 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:07.732861 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:07.733161 10791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:07.736378 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "bce17920f39c47b2b1ea398334bf61c0"
format_stamp: "Formatted at 2026-09-28 10:44:07 on dist-test-slave-mn38"
server_key: "d466efd21dde39ad6bf111d52d227981"
server_key_iv: "20e323bd6eaa3fefb4015adf04935f79"
server_key_version: "encryptionkey@0"
I20260928 10:44:07.737186 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "bce17920f39c47b2b1ea398334bf61c0"
format_stamp: "Formatted at 2026-09-28 10:44:07 on dist-test-slave-mn38"
server_key: "d466efd21dde39ad6bf111d52d227981"
server_key_iv: "20e323bd6eaa3fefb4015adf04935f79"
server_key_version: "encryptionkey@0"
I20260928 10:44:07.743209 10791 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 10:44:07.747084 10806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:07.748749 10791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:44:07.748973 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "bce17920f39c47b2b1ea398334bf61c0"
format_stamp: "Formatted at 2026-09-28 10:44:07 on dist-test-slave-mn38"
server_key: "d466efd21dde39ad6bf111d52d227981"
server_key_iv: "20e323bd6eaa3fefb4015adf04935f79"
server_key_version: "encryptionkey@0"
I20260928 10:44:07.749234 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:07.764125 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:07.769261 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:07.769587 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:07.793803 10791 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:35751
I20260928 10:44:07.793817 10867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:35751 every 8 connection(s)
I20260928 10:44:07.795346 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:07.800429 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:07.800855 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 10791
I20260928 10:44:07.801082 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:07.801798 9067 external_mini_cluster.cc:1428] Setting key fe4cc5f837f4138741db3bff070853ab
I20260928 10:44:07.816385 10868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: Bootstrap starting.
Sep 28 10:44:07 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:07.820640 10868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:07.822016 10868 log.cc:824] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:07.825356 10868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: No bootstrap required, opened a new log
I20260928 10:44:07.829022 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:07.804110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.8.218.254:35751)
0928 10:44:07.805073 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:07.805087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:07.805135 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:44:07.807396 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:07.807436 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:07.808042 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:07.808504 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:07.810548 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:07.812193 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:07.814194 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:07.815160 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:07.815332 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:07.820536 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:07.820547 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:07.820565 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:07.820614 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:07.824141 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:07.825003 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:07.825011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:07.825021 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:07.825233 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:07.825706 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:07.825711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:07.825713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:07.826367 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:07.826688 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:07.827192 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 10:44:07.827727 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":429,"threads_started":1}
I20260928 10:44:07.834303 10868 raft_consensus.cc:348] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce17920f39c47b2b1ea398334bf61c0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 35751 } }
I20260928 10:44:07.834769 10868 raft_consensus.cc:374] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:07.834872 10868 raft_consensus.cc:729] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce17920f39c47b2b1ea398334bf61c0, State: Initialized, Role: FOLLOWER
I20260928 10:44:07.835503 10868 consensus_queue.cc:260] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce17920f39c47b2b1ea398334bf61c0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 35751 } }
I20260928 10:44:07.835753 10868 raft_consensus.cc:388] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:07.835891 10868 raft_consensus.cc:482] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:07.836066 10868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:07.848762 10868 raft_consensus.cc:504] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce17920f39c47b2b1ea398334bf61c0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 35751 } }
I20260928 10:44:07.849871 10868 leader_election.cc:302] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bce17920f39c47b2b1ea398334bf61c0; no voters:
I20260928 10:44:07.850835 10868 leader_election.cc:288] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:07.850909 10873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:07.852249 10873 raft_consensus.cc:686] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [term 1 LEADER]: Becoming Leader. State: Replica: bce17920f39c47b2b1ea398334bf61c0, State: Running, Role: LEADER
I20260928 10:44:07.852890 10873 consensus_queue.cc:237] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce17920f39c47b2b1ea398334bf61c0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 35751 } }
I20260928 10:44:07.853395 10868 sys_catalog.cc:564] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:07.856132 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bce17920f39c47b2b1ea398334bf61c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce17920f39c47b2b1ea398334bf61c0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 35751 } } }
I20260928 10:44:07.856523 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:07.856102 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bce17920f39c47b2b1ea398334bf61c0. Latest consensus state: current_term: 1 leader_uuid: "bce17920f39c47b2b1ea398334bf61c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce17920f39c47b2b1ea398334bf61c0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 35751 } } }
I20260928 10:44:07.858381 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:07.859050 10878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:07.864542 10878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:07.875640 10878 catalog_manager.cc:1347] Generated new cluster ID: 3b6e2054f0f04777834abdecb65025bf
I20260928 10:44:07.875809 10878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:07.894970 10878 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:07.896461 10878 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:07.904608 10878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: Generated new TSK 0
I20260928 10:44:07.905862 10878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
W20260928 10:44:08.922956 9067 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.218.193: real 1.000s user 0.002s sys 0.013s
I20260928 10:44:08.923969 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:35751
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:09.293321 10894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:09.293672 10894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:09.293782 10894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:09.302163 10894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:09.302309 10894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:09.302399 10894 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:44:09.302452 10894 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:09.302523 10894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:09.314422 10894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:35751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:09.316740 10894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:09.318288 10894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:09.333335 10902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:09.333307 10900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:09.333281 10899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:09.334239 10894 server_base.cc:1034] running on GCE node
I20260928 10:44:09.334906 10894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:09.336199 10894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:09.337563 10894 hybrid_clock.cc:648] HybridClock initialized: now 1790592249337500 us; error 50 us; skew 500 ppm
I20260928 10:44:09.337950 10894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:09 dist-test-slave-mn38 krb5kdc[10778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592249, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:09.341364 10894 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:44:09.343008 10894 webserver.cc:460] Webserver started at http://127.8.218.193:45551/ using document root <none> and password file <none>
I20260928 10:44:09.343753 10894 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:09.343868 10894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:09.344149 10894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:09.347438 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "29731b7a843a49e9a9e0a86f0664d399"
format_stamp: "Formatted at 2026-09-28 10:44:09 on dist-test-slave-mn38"
server_key: "7596bad7e5f6dbf688faafb1037d8edd"
server_key_iv: "6f192a519f6bd983120c14fe152ea973"
server_key_version: "encryptionkey@0"
I20260928 10:44:09.348294 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "29731b7a843a49e9a9e0a86f0664d399"
format_stamp: "Formatted at 2026-09-28 10:44:09 on dist-test-slave-mn38"
server_key: "7596bad7e5f6dbf688faafb1037d8edd"
server_key_iv: "6f192a519f6bd983120c14fe152ea973"
server_key_version: "encryptionkey@0"
I20260928 10:44:09.354519 10894 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:44:09.358508 10909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:09.360078 10894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:44:09.360232 10894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "29731b7a843a49e9a9e0a86f0664d399"
format_stamp: "Formatted at 2026-09-28 10:44:09 on dist-test-slave-mn38"
server_key: "7596bad7e5f6dbf688faafb1037d8edd"
server_key_iv: "6f192a519f6bd983120c14fe152ea973"
server_key_version: "encryptionkey@0"
I20260928 10:44:09.360423 10894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:09.378029 10894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:09.382776 10894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:09.383054 10894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:09.385991 10894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:09.386133 10894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:09.386253 10894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:09.386373 10894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:09 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592249, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:09.414251 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:09.387725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:35751 (local address 127.8.218.193:45499)
0928 10:44:09.389063 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:09.389122 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:09.391452 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:09.392361 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:09.392371 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:09.393176 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:09.394223 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:09.394248 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.395976 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.395984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:09.398247 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:09.398265 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.399235 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.399246 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:09.399617 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:09.400578 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:09.400619 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:09.404482 (+ 3863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:09.408556 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:09.408575 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:09.408603 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:09.409376 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:09.409977 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:09.409982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:09.409990 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:09.410357 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:09.411095 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:09.411101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:09.411695 (+ 594us) client_negotiation.cc:769] Sending connection context
0928 10:44:09.412077 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 10:44:09.412712 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":410,"threads_started":1}
I20260928 10:44:09.414408 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:09.386965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:45499 (local address 127.8.218.254:35751)
0928 10:44:09.387404 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:09.387409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:09.389282 (+ 1873us) server_negotiation.cc:407] Connection header received
0928 10:44:09.391840 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:09.391847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:09.391946 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:09.392045 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:09.394557 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.395695 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.398449 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.398929 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.399079 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:09.404885 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:09.404894 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:09.404901 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:09.404956 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:09.408305 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:09.409627 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:09.409634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:09.409638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:09.409844 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:09.410497 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:09.410502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:09.410504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:09.410874 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:09.411026 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:09.413655 (+ 2629us) server_negotiation.cc:299] Negotiation successful
0928 10:44:09.413984 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":228,"threads_started":1}
I20260928 10:44:09.429823 10894 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:34153
I20260928 10:44:09.429841 11029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:34153 every 8 connection(s)
I20260928 10:44:09.431677 10894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:09.434794 11030 heartbeater.cc:347] Connected to a master server at 127.8.218.254:35751
I20260928 10:44:09.435091 11030 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:09.435705 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 10894
I20260928 10:44:09.435788 11030 heartbeater.cc:511] Master 127.8.218.254:35751 requested a full tablet report, sending...
I20260928 10:44:09.435938 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:09.436719 9067 external_mini_cluster.cc:1428] Setting key 5fbc90fdcfdcf1dca2d0859b2957a4f7
I20260928 10:44:09.437127 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:09.422066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:35751 (local address 127.8.218.193:36535)
0928 10:44:09.422774 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:09.422798 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:09.422997 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:09.423407 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:09.423412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:09.423912 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:09.424231 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:09.424243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.425580 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.425586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:09.426663 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:09.426679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.427631 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.427642 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:09.427852 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:09.428657 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:09.428682 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:09.429976 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:09.434076 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:09.434083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:09.434087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:09.434649 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:09.435066 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:09.435070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:09.435073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:09.435227 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:09.435981 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:09.435986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:09.436188 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 10:44:09.436332 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 10:44:09.436699 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":356,"threads_started":1}
I20260928 10:44:09.438328 10823 ts_manager.cc:194] Registered new tserver with Master: 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153)
I20260928 10:44:09.439209 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:09.422070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:36535 (local address 127.8.218.254:35751)
0928 10:44:09.422277 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:09.422283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:09.422916 (+ 633us) server_negotiation.cc:407] Connection header received
0928 10:44:09.423142 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:09.423148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:09.423238 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:09.423339 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:09.424413 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.425408 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.426843 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.427387 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.427535 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:09.430258 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:09.430265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:09.430269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:09.430313 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:09.433911 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:09.434774 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:09.434778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:09.434781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:09.434926 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:09.435394 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:09.435400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:09.435403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:09.435765 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:09.435916 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:09.437632 (+ 1716us) server_negotiation.cc:299] Negotiation successful
0928 10:44:09.438857 (+ 1225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:44:09.440358 10823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:45499
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:44:09.485639 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:35751
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:09.847522 11034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:09.847853 11034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:09.847961 11034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:09.855945 11034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:09.856091 11034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:09.856179 11034 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:44:09.856232 11034 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:09.856293 11034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:09.866926 11034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:35751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:09.868700 11034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:09.870220 11034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:09.884254 11039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:09.884318 11042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:09.884353 11040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:09.885335 11034 server_base.cc:1034] running on GCE node
I20260928 10:44:09.886116 11034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:09.887184 11034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:09.888422 11034 hybrid_clock.cc:648] HybridClock initialized: now 1790592249888373 us; error 49 us; skew 500 ppm
I20260928 10:44:09.888775 11034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:09 dist-test-slave-mn38 krb5kdc[10778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592249, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:09.892433 11034 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:44:09.894145 11034 webserver.cc:460] Webserver started at http://127.8.218.194:34619/ using document root <none> and password file <none>
I20260928 10:44:09.894912 11034 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:09.895042 11034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:09.895354 11034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:09.898626 11034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "4c478f01b25449e29d54fc4879d939db"
format_stamp: "Formatted at 2026-09-28 10:44:09 on dist-test-slave-mn38"
server_key: "706d72b104f577c3527805f1a4c3c295"
server_key_iv: "7d004c09226dafcce64a5c2a1aad3e6c"
server_key_version: "encryptionkey@0"
I20260928 10:44:09.899479 11034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "4c478f01b25449e29d54fc4879d939db"
format_stamp: "Formatted at 2026-09-28 10:44:09 on dist-test-slave-mn38"
server_key: "706d72b104f577c3527805f1a4c3c295"
server_key_iv: "7d004c09226dafcce64a5c2a1aad3e6c"
server_key_version: "encryptionkey@0"
I20260928 10:44:09.905792 11034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:44:09.910986 11049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:09.912761 11034 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:44:09.912981 11034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "4c478f01b25449e29d54fc4879d939db"
format_stamp: "Formatted at 2026-09-28 10:44:09 on dist-test-slave-mn38"
server_key: "706d72b104f577c3527805f1a4c3c295"
server_key_iv: "7d004c09226dafcce64a5c2a1aad3e6c"
server_key_version: "encryptionkey@0"
I20260928 10:44:09.913213 11034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:09.933023 11034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:09.938086 11034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:09.938390 11034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:09.941295 11034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:09.941408 11034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:09.941604 11034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:09.941686 11034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:09 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592249, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:09.971045 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:09.941699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:45727 (local address 127.8.218.254:35751)
0928 10:44:09.942233 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:09.942239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:09.943740 (+ 1501us) server_negotiation.cc:407] Connection header received
0928 10:44:09.946722 (+ 2982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:09.946729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:09.946833 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:09.946962 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:09.949491 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.950517 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.953688 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.954277 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.954425 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:09.960614 (+ 6189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:09.960623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:09.960628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:09.960674 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:09.964558 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:09.966315 (+ 1757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:09.966323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:09.966328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:09.966571 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:09.967468 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:09.967475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:09.967480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:09.968412 (+ 932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:09.968668 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:09.970208 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0928 10:44:09.970564 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":228,"threads_started":1}
I20260928 10:44:09.972765 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:09.942641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:35751 (local address 127.8.218.194:45727)
0928 10:44:09.943538 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:09.943591 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:09.946297 (+ 2706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:09.947340 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:09.947355 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:09.948245 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:09.949218 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:09.949249 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.950727 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.950732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:09.953477 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:09.953501 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.954560 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.954568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:09.954923 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:09.955970 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:09.956005 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:09.960225 (+ 4220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:09.964813 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:09.964825 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:09.964855 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:09.966054 (+ 1199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:09.966798 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:09.966804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:09.966809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:09.967268 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:09.968685 (+ 1417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:09.968702 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:09.969553 (+ 851us) client_negotiation.cc:769] Sending connection context
0928 10:44:09.970052 (+ 499us) client_negotiation.cc:240] Negotiation successful
0928 10:44:09.970915 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":346,"threads_started":1}
I20260928 10:44:09.981316 11034 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:35805
I20260928 10:44:09.981359 11168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:35805 every 8 connection(s)
I20260928 10:44:09.983076 11034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:09.985364 11170 heartbeater.cc:347] Connected to a master server at 127.8.218.254:35751
I20260928 10:44:09.985687 11170 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:09.986440 11170 heartbeater.cc:511] Master 127.8.218.254:35751 requested a full tablet report, sending...
I20260928 10:44:09.986646 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 11034
I20260928 10:44:09.986847 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:09.987286 9067 external_mini_cluster.cc:1428] Setting key 5a47589b2edf5de978522fdb8ee9e8bf
I20260928 10:44:09.988039 10823 ts_manager.cc:194] Registered new tserver with Master: 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:09.988993 10823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:45727
I20260928 10:44:09.995461 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:09.981549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:44265 (local address 127.8.218.254:35751)
0928 10:44:09.981745 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:09.981750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:09.982464 (+ 714us) server_negotiation.cc:407] Connection header received
0928 10:44:09.982643 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:09.982649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:09.982723 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:09.982805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:09.984063 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.985032 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.986770 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.987408 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.987552 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:09.989703 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:09.989710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:09.989715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:09.989753 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:09.992588 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:09.993362 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:09.993367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:09.993370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:09.993553 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:09.993944 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:09.993950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:09.993954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:09.994295 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:09.994451 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:09.994866 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 10:44:09.995099 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:44:09.995522 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:09.981657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:35751 (local address 127.8.218.194:44265)
0928 10:44:09.982341 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:09.982364 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:09.982517 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:09.982856 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:09.982860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:09.983382 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:09.983861 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:09.983878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.985187 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.985193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:09.986590 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:09.986609 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:09.987611 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:09.987618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:09.987792 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:09.988498 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:09.988521 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:09.989511 (+ 990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:09.992776 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:09.992782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:09.992787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:09.993211 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:09.993668 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:09.993673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:09.993677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:09.993825 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:09.994429 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:09.994434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:09.994613 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 10:44:09.994731 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 10:44:09.995089 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":411,"threads_started":1}
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:44:10.033605 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:35751
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:10.400894 11174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:10.401217 11174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:10.401352 11174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:10.409047 11174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:10.409189 11174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:10.409296 11174 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 10:44:10.409366 11174 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:10.409499 11174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:10.420862 11174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:35751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:10.422410 11174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:10.423884 11174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:10.439623 11179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:10.439688 11180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:10.439654 11182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:10.440516 11174 server_base.cc:1034] running on GCE node
I20260928 10:44:10.441203 11174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:10.442267 11174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:10.443481 11174 hybrid_clock.cc:648] HybridClock initialized: now 1790592250443430 us; error 48 us; skew 500 ppm
I20260928 10:44:10.443508 11030 heartbeater.cc:503] Master 127.8.218.254:35751 was elected leader, sending a full tablet report...
I20260928 10:44:10.443820 11174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:10 dist-test-slave-mn38 krb5kdc[10778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592250, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:10.447414 11174 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:44:10.449507 11174 webserver.cc:460] Webserver started at http://127.8.218.195:40873/ using document root <none> and password file <none>
I20260928 10:44:10.450508 11174 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:10.450657 11174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:10.450979 11174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:10.454206 11174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "e0b2baa0c0424c25a3fee454e2c121f0"
format_stamp: "Formatted at 2026-09-28 10:44:10 on dist-test-slave-mn38"
server_key: "199d60d358a8588c663773889b25069e"
server_key_iv: "574f44c57aebd27bbdac550a3ae96b1d"
server_key_version: "encryptionkey@0"
I20260928 10:44:10.455032 11174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "e0b2baa0c0424c25a3fee454e2c121f0"
format_stamp: "Formatted at 2026-09-28 10:44:10 on dist-test-slave-mn38"
server_key: "199d60d358a8588c663773889b25069e"
server_key_iv: "574f44c57aebd27bbdac550a3ae96b1d"
server_key_version: "encryptionkey@0"
I20260928 10:44:10.460798 11174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 10:44:10.465204 11189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:10.467046 11174 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:10.467258 11174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "e0b2baa0c0424c25a3fee454e2c121f0"
format_stamp: "Formatted at 2026-09-28 10:44:10 on dist-test-slave-mn38"
server_key: "199d60d358a8588c663773889b25069e"
server_key_iv: "574f44c57aebd27bbdac550a3ae96b1d"
server_key_version: "encryptionkey@0"
I20260928 10:44:10.467507 11174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:10.500773 11174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:10.505926 11174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:10.506258 11174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:10.509232 11174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:10.509366 11174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:10.509516 11174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:10.509611 11174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:10 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592250, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:10.540122 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.511206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:35751 (local address 127.8.218.195:41195)
0928 10:44:10.512325 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:10.512400 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:10.516146 (+ 3746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:10.517347 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:10.517357 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:10.518258 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:10.519451 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.519488 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.521209 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.521216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.523404 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.523429 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.524753 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.524758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.525014 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.525888 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:10.525918 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:10.530436 (+ 4518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:10.534449 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:10.534463 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:10.534482 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:10.535077 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:10.535552 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:10.535558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:10.535562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:10.535839 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:10.536597 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:10.536611 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:10.537176 (+ 565us) client_negotiation.cc:769] Sending connection context
0928 10:44:10.537568 (+ 392us) client_negotiation.cc:240] Negotiation successful
0928 10:44:10.538244 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":520,"threads_started":1}
I20260928 10:44:10.541733 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.510463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:41195 (local address 127.8.218.254:35751)
0928 10:44:10.510916 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:10.510920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:10.512568 (+ 1648us) server_negotiation.cc:407] Connection header received
0928 10:44:10.516583 (+ 4015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:10.516589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:10.516696 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:10.516805 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:10.519733 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.520979 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.523707 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.524467 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.524629 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.530828 (+ 6199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:10.530835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:10.530838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:10.530879 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:10.534248 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:10.535247 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:10.535255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:10.535260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:10.535443 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:10.536024 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:10.536030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:10.536033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:10.536379 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:10.540814 (+ 4435us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:10.541093 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 10:44:10.541386 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260928 10:44:10.559633 11174 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:39885
I20260928 10:44:10.559706 11309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:39885 every 8 connection(s)
I20260928 10:44:10.561056 11174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:10.563253 11310 heartbeater.cc:347] Connected to a master server at 127.8.218.254:35751
I20260928 10:44:10.563618 11310 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:10.564447 11310 heartbeater.cc:511] Master 127.8.218.254:35751 requested a full tablet report, sending...
I20260928 10:44:10.565625 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 11174
I20260928 10:44:10.565812 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:10.566443 9067 external_mini_cluster.cc:1428] Setting key 33b74af9728272a64c1d59a2b10f2cb4
I20260928 10:44:10.568050 10822 ts_manager.cc:194] Registered new tserver with Master: e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195:39885)
I20260928 10:44:10.568984 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.552524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:35751 (local address 127.8.218.195:43089)
0928 10:44:10.553740 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:10.553763 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:10.553978 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:10.554415 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:10.554424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:10.555145 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:10.555480 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.555493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.557170 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.557177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.558922 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.558945 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.560142 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.560147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.560337 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.560932 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:10.560949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:10.561995 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:10.565997 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:10.566003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:10.566007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:10.566527 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:10.566963 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:10.566969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:10.566972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:10.567125 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:10.567869 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:10.567873 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:10.568069 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 10:44:10.568206 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 10:44:10.568545 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":775,"threads_started":1}
I20260928 10:44:10.569235 10822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:41195
I20260928 10:44:10.569904 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.552724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:43089 (local address 127.8.218.254:35751)
0928 10:44:10.552936 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:10.552944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:10.553876 (+ 932us) server_negotiation.cc:407] Connection header received
0928 10:44:10.554125 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:10.554129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:10.554199 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:10.554362 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:10.555691 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.556929 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.559119 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.559793 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.559930 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.562266 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:10.562273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:10.562278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:10.562319 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:10.565855 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:10.566662 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:10.566667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:10.566669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:10.566825 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:10.567257 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:10.567261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:10.567263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:10.567626 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:10.567805 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:10.569316 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0928 10:44:10.569578 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 10:44:10.570778 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:44:10 dist-test-slave-mn38 krb5kdc[10778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:44:10 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:10.621717 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.606105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.8.218.254:35751)
0928 10:44:10.606422 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:10.606428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:10.606539 (+ 111us) server_negotiation.cc:407] Connection header received
0928 10:44:10.606790 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:10.606941 (+ 151us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:10.607041 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:10.607155 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:10.608746 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.609539 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.610924 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.611373 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.611494 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.615724 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:10.615730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:10.615734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:10.615781 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:10.618764 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:10.619546 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:10.619550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:10.619553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:10.619698 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:10.620110 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:10.620114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:10.620117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:10.620518 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:10.620762 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:10.621112 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 10:44:10.621350 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 10:44:10.626930 10823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:10.636236 10823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:10.660786 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.649629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34153 (local address 127.0.0.1:44234)
0928 10:44:10.651208 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:10.651271 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:10.651579 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:10.652497 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:10.652510 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:10.652548 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:10.653051 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.653067 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.655694 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.655698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.658133 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.658146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.659721 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.659727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.659927 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.659979 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 10:44:10.660125 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 10:44:10.660336 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"mutex_wait_us":4,"thread_start_us":2975,"threads_started":1}
I20260928 10:44:10.660899 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.650943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.8.218.195:39885)
0928 10:44:10.652437 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:10.652446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:10.653154 (+ 708us) server_negotiation.cc:407] Connection header received
0928 10:44:10.653256 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:10.653263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:10.653632 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:10.653819 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:10.654708 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.656247 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.658400 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.659879 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.659979 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.660200 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:10.660381 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 10:44:10.660554 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":1149,"threads_started":1}
I20260928 10:44:10.661269 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.649777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.8.218.193:34153)
0928 10:44:10.650571 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:10.650583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:10.651396 (+ 813us) server_negotiation.cc:407] Connection header received
0928 10:44:10.651828 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:10.651838 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:10.652137 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:10.652339 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:10.653258 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.655563 (+ 2305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.658341 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.659508 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.659639 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.659863 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:10.660625 (+ 762us) server_negotiation.cc:299] Negotiation successful
0928 10:44:10.660816 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":438,"threads_started":1}
I20260928 10:44:10.661574 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.650261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:39885 (local address 127.0.0.1:54490)
0928 10:44:10.652908 (+ 2647us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:10.652928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:10.653052 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:10.653942 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:10.653946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:10.653972 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:10.654481 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.654494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.656375 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.656380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.658201 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.658220 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.660053 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.660059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.660207 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.660236 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 10:44:10.660323 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 10:44:10.660434 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2492,"mutex_wait_us":32,"thread_start_us":514,"threads_started":1}
I20260928 10:44:10.662359 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.653008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35805 (local address 127.0.0.1:41900)
0928 10:44:10.654602 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:10.654619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:10.654726 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:10.655361 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:10.655365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:10.655391 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:10.655746 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.655758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.657543 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.657548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.659666 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:10.659682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.661659 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.661666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:10.661858 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.661889 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:44:10.661949 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 10:44:10.662074 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1432,"thread_start_us":1270,"threads_started":1}
I20260928 10:44:10.663398 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:10.653226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.8.218.194:35805)
0928 10:44:10.653829 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:10.653838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:10.654837 (+ 999us) server_negotiation.cc:407] Connection header received
0928 10:44:10.654945 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:10.654952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:10.655177 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:10.655378 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:10.655962 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.657375 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.659873 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:10.661258 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:10.661422 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:10.662868 (+ 1446us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:10.662960 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 10:44:10.663134 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":268,"threads_started":1}
I20260928 10:44:10.665719 11240 tablet_service.cc:1467] Processing CreateTablet for tablet 33faebecfb5a419da9a313eb8e520c2c (DEFAULT_TABLE table=test-table [id=da78d09197424f9a8d1349373845fc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:10.665720 10960 tablet_service.cc:1467] Processing CreateTablet for tablet 33faebecfb5a419da9a313eb8e520c2c (DEFAULT_TABLE table=test-table [id=da78d09197424f9a8d1349373845fc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:10.667425 11240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33faebecfb5a419da9a313eb8e520c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:10.667425 10960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33faebecfb5a419da9a313eb8e520c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:10.667263 11100 tablet_service.cc:1467] Processing CreateTablet for tablet 33faebecfb5a419da9a313eb8e520c2c (DEFAULT_TABLE table=test-table [id=da78d09197424f9a8d1349373845fc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:10.669334 11100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33faebecfb5a419da9a313eb8e520c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:10.684371 11330 tablet_bootstrap.cc:492] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Bootstrap starting.
I20260928 10:44:10.689910 11330 tablet_bootstrap.cc:654] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:10.691753 11330 log.cc:824] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:10.693320 11329 tablet_bootstrap.cc:492] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Bootstrap starting.
I20260928 10:44:10.696961 11329 tablet_bootstrap.cc:654] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:10.698285 11331 tablet_bootstrap.cc:492] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Bootstrap starting.
I20260928 10:44:10.698480 11329 log.cc:824] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:10.700932 11330 tablet_bootstrap.cc:492] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: No bootstrap required, opened a new log
I20260928 10:44:10.701329 11330 ts_tablet_manager.cc:1397] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260928 10:44:10.703791 11331 tablet_bootstrap.cc:654] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:10.704977 11329 tablet_bootstrap.cc:492] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: No bootstrap required, opened a new log
I20260928 10:44:10.705485 11329 ts_tablet_manager.cc:1397] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260928 10:44:10.705827 11331 log.cc:824] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:10.708657 11331 tablet_bootstrap.cc:492] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: No bootstrap required, opened a new log
I20260928 10:44:10.708956 11331 ts_tablet_manager.cc:1397] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 10:44:10.713675 11330 raft_consensus.cc:348] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:10.714577 11330 raft_consensus.cc:374] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:10.714704 11330 raft_consensus.cc:729] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Initialized, Role: FOLLOWER
I20260928 10:44:10.715762 11330 consensus_queue.cc:260] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:10.717386 11329 raft_consensus.cc:348] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:10.717940 11329 raft_consensus.cc:374] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:10.718037 11329 raft_consensus.cc:729] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c478f01b25449e29d54fc4879d939db, State: Initialized, Role: FOLLOWER
I20260928 10:44:10.717257 11331 raft_consensus.cc:348] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:10.718202 11331 raft_consensus.cc:374] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:10.718370 11331 raft_consensus.cc:729] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29731b7a843a49e9a9e0a86f0664d399, State: Initialized, Role: FOLLOWER
I20260928 10:44:10.718796 11329 consensus_queue.cc:260] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:10.718914 11331 consensus_queue.cc:260] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:10.720697 11331 ts_tablet_manager.cc:1428] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 10:44:10.721676 11170 heartbeater.cc:503] Master 127.8.218.254:35751 was elected leader, sending a full tablet report...
I20260928 10:44:10.722427 11329 ts_tablet_manager.cc:1428] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260928 10:44:10.723451 11310 heartbeater.cc:503] Master 127.8.218.254:35751 was elected leader, sending a full tablet report...
I20260928 10:44:10.724212 11330 ts_tablet_manager.cc:1428] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Time spent starting tablet: real 0.023s user 0.013s sys 0.008s
W20260928 10:44:10.735608 11171 tablet.cc:2369] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:44:10.813732 11311 tablet.cc:2369] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:44:10.934952 11031 tablet.cc:2369] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:44:11.012360 11335 raft_consensus.cc:482] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:11.012688 11335 raft_consensus.cc:504] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.014483 11335 leader_election.cc:288] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:11.020411 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.014915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34153 (local address 127.8.218.195:53915)
0928 10:44:11.015089 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:11.015111 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:11.015244 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:11.015737 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:11.015741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:11.015767 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:11.015940 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.015950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.017014 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.017020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.018516 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.018530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.019641 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.019647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.019847 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.019883 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:44:11.019986 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 10:44:11.020133 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 10:44:11.020450 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.015045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:53915 (local address 127.8.218.193:34153)
0928 10:44:11.015256 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.015262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.015284 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:44:11.015354 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.015359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.015432 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.015516 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:11.016088 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.016901 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.018697 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.019490 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.019646 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.019735 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.020045 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.020208 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:44:11.021752 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.015291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35805 (local address 127.8.218.195:45835)
0928 10:44:11.015874 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:11.015892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:11.016020 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:11.016324 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:11.016328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:11.016350 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:11.016663 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.016672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.017855 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.017859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.019308 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.019321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.020972 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.020978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.021145 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.021178 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 10:44:11.021324 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 10:44:11.021498 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":311,"threads_started":1}
I20260928 10:44:11.021818 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.015417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:45835 (local address 127.8.218.194:35805)
0928 10:44:11.015617 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.015623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.015982 (+ 359us) server_negotiation.cc:407] Connection header received
0928 10:44:11.016150 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.016154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.016215 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.016297 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:11.016808 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.017715 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.019472 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.020614 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.020772 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.020827 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.021381 (+ 554us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.021554 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:44:11.021775 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33faebecfb5a419da9a313eb8e520c2c" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399" is_pre_election: true
I20260928 10:44:11.022440 10980 raft_consensus.cc:2454] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 0.
I20260928 10:44:11.022868 11120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33faebecfb5a419da9a313eb8e520c2c" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c478f01b25449e29d54fc4879d939db" is_pre_election: true
I20260928 10:44:11.023171 11190 leader_election.cc:302] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29731b7a843a49e9a9e0a86f0664d399, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:11.023423 11120 raft_consensus.cc:2454] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 0.
I20260928 10:44:11.023608 11335 raft_consensus.cc:2781] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:11.023743 11335 raft_consensus.cc:482] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:11.023854 11335 raft_consensus.cc:3037] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.026539 11335 raft_consensus.cc:504] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.027238 11335 leader_election.cc:288] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 election: Requested vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:11.028008 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33faebecfb5a419da9a313eb8e520c2c" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399"
I20260928 10:44:11.028009 11120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33faebecfb5a419da9a313eb8e520c2c" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c478f01b25449e29d54fc4879d939db"
I20260928 10:44:11.028254 11120 raft_consensus.cc:3037] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.028245 10980 raft_consensus.cc:3037] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.031600 11120 raft_consensus.cc:2454] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 1.
I20260928 10:44:11.031600 10980 raft_consensus.cc:2454] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 1.
I20260928 10:44:11.032140 11190 leader_election.cc:302] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c478f01b25449e29d54fc4879d939db, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:11.032503 11335 raft_consensus.cc:2781] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:11.032990 11335 raft_consensus.cc:686] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 LEADER]: Becoming Leader. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Running, Role: LEADER
I20260928 10:44:11.033535 11335 consensus_queue.cc:237] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.038168 10823 catalog_manager.cc:5495] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } health_report { overall_health: HEALTHY } } }
Sep 28 10:44:11 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592247, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:11.150622 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.135863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:35751 (local address 127.0.0.1:54884)
0928 10:44:11.136767 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:11.136785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:11.136961 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:11.137340 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:11.137345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:11.138155 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:11.138438 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.138449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.139608 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.139613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.140617 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.140630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.141340 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.141350 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.141606 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.142268 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:11.142353 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:11.144631 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:11.148006 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:11.148020 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:11.148027 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:11.148500 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:11.148894 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:11.148903 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:11.148906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:11.149057 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:11.149779 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:11.149787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:11.149945 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 10:44:11.150087 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 10:44:11.150312 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":642,"threads_started":1}
I20260928 10:44:11.150736 11346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.135878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.8.218.254:35751)
0928 10:44:11.136507 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.136514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.136890 (+ 376us) server_negotiation.cc:407] Connection header received
0928 10:44:11.137142 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.137146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.137203 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.137293 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:11.138616 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.139481 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.140755 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.141152 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.141257 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.144887 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:11.144893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:11.144897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:11.144931 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:11.147847 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:11.148621 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:11.148626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:11.148628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:11.148761 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:11.149194 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:11.149198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:11.149200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:11.149577 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:11.149793 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.150242 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.150434 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":266,"threads_started":1}
I20260928 10:44:11.156224 10823 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:54884:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:44:11.157289 10823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:11.164858 10960 tablet_service.cc:1467] Processing CreateTablet for tablet 10ccf715075742f98e7cac7a818cb26b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f198725a6a8f486993d3f9c7f1f55329]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:44:11.164808 11100 tablet_service.cc:1467] Processing CreateTablet for tablet 10ccf715075742f98e7cac7a818cb26b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f198725a6a8f486993d3f9c7f1f55329]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:44:11.166077 11100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ccf715075742f98e7cac7a818cb26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:11.166076 10960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ccf715075742f98e7cac7a818cb26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:11.170809 11240 tablet_service.cc:1467] Processing CreateTablet for tablet 10ccf715075742f98e7cac7a818cb26b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f198725a6a8f486993d3f9c7f1f55329]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:44:11.172034 11240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ccf715075742f98e7cac7a818cb26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:11.172287 11331 tablet_bootstrap.cc:492] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: Bootstrap starting.
I20260928 10:44:11.173728 11329 tablet_bootstrap.cc:492] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: Bootstrap starting.
I20260928 10:44:11.173843 11331 tablet_bootstrap.cc:654] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:11.175415 11329 tablet_bootstrap.cc:654] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:11.175976 11331 tablet_bootstrap.cc:492] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: No bootstrap required, opened a new log
I20260928 10:44:11.176087 11331 ts_tablet_manager.cc:1397] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 10:44:11.177007 11331 raft_consensus.cc:348] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.177278 11331 raft_consensus.cc:374] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:11.177407 11331 raft_consensus.cc:729] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29731b7a843a49e9a9e0a86f0664d399, State: Initialized, Role: FOLLOWER
I20260928 10:44:11.177685 11331 consensus_queue.cc:260] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.178076 11329 tablet_bootstrap.cc:492] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: No bootstrap required, opened a new log
I20260928 10:44:11.178244 11329 ts_tablet_manager.cc:1397] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260928 10:44:11.178265 11331 ts_tablet_manager.cc:1428] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:44:11.178184 11336 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.178612 11336 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:11.179004 11329 raft_consensus.cc:348] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.179241 11329 raft_consensus.cc:374] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:11.179328 11329 raft_consensus.cc:729] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c478f01b25449e29d54fc4879d939db, State: Initialized, Role: FOLLOWER
I20260928 10:44:11.179574 11329 consensus_queue.cc:260] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.180034 11330 tablet_bootstrap.cc:492] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: Bootstrap starting.
I20260928 10:44:11.180189 11329 ts_tablet_manager.cc:1428] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 10:44:11.180110 11337 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.180527 11337 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:11.181902 11330 tablet_bootstrap.cc:654] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:11.187978 11330 tablet_bootstrap.cc:492] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: No bootstrap required, opened a new log
I20260928 10:44:11.188143 11330 ts_tablet_manager.cc:1397] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 10:44:11.188815 11330 raft_consensus.cc:348] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.189016 11330 raft_consensus.cc:374] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:11.189098 11330 raft_consensus.cc:729] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Initialized, Role: FOLLOWER
I20260928 10:44:11.189307 11330 consensus_queue.cc:260] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.189924 11330 ts_tablet_manager.cc:1428] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 10:44:11.189819 11335 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.190147 11335 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:11.329701 11337 raft_consensus.cc:482] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:11.329941 11337 raft_consensus.cc:504] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.331596 11337 leader_election.cc:288] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195:39885)
I20260928 10:44:11.340543 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.332110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:34525 (local address 127.8.218.193:34153)
0928 10:44:11.332316 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.332339 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.334166 (+ 1827us) server_negotiation.cc:407] Connection header received
0928 10:44:11.334516 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.334522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.334608 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.334694 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:11.335544 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.336311 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.338537 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.339405 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.339606 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.339699 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.340063 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.340222 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:44:11.340561 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.332124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:50461 (local address 127.8.218.195:39885)
0928 10:44:11.333585 (+ 1461us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.333592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.334418 (+ 826us) server_negotiation.cc:407] Connection header received
0928 10:44:11.334748 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.334755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.334827 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.334916 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:11.335795 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.336498 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.338527 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.339451 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.339629 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.339708 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.340067 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.340228 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":1116,"threads_started":1}
I20260928 10:44:11.340735 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.331877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34153 (local address 127.8.218.194:34525)
0928 10:44:11.334022 (+ 2145us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:11.334051 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:11.334306 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:11.334718 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:11.334722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:11.334750 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:11.335333 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.335351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.336457 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.336462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.338337 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.338352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.339705 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.339712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.339873 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.339912 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:44:11.340011 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 10:44:11.340237 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2004,"mutex_wait_us":128,"thread_start_us":2006,"threads_started":1}
I20260928 10:44:11.340797 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.332050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:39885 (local address 127.8.218.194:50461)
0928 10:44:11.334300 (+ 2250us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:11.334315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:11.334543 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:11.335075 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:11.335082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:11.335102 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:11.335602 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.335614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.336633 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.336637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.338315 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.338334 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.339754 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.339761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.339906 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.339930 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 10:44:11.340019 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 10:44:11.340244 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2132,"mutex_wait_us":163,"thread_start_us":1711,"threads_started":1}
I20260928 10:44:11.341723 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ccf715075742f98e7cac7a818cb26b" candidate_uuid: "4c478f01b25449e29d54fc4879d939db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399" is_pre_election: true
I20260928 10:44:11.342104 10980 raft_consensus.cc:2454] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c478f01b25449e29d54fc4879d939db in term 0.
I20260928 10:44:11.342324 11260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ccf715075742f98e7cac7a818cb26b" candidate_uuid: "4c478f01b25449e29d54fc4879d939db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" is_pre_election: true
I20260928 10:44:11.342707 11050 leader_election.cc:302] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29731b7a843a49e9a9e0a86f0664d399, 4c478f01b25449e29d54fc4879d939db; no voters:
I20260928 10:44:11.342809 11260 raft_consensus.cc:2454] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c478f01b25449e29d54fc4879d939db in term 0.
I20260928 10:44:11.343155 11337 raft_consensus.cc:2781] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:11.343290 11337 raft_consensus.cc:482] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:11.343393 11337 raft_consensus.cc:3037] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.345836 11337 raft_consensus.cc:504] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.346566 11337 leader_election.cc:288] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [CANDIDATE]: Term 1 election: Requested vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195:39885)
I20260928 10:44:11.347225 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ccf715075742f98e7cac7a818cb26b" candidate_uuid: "4c478f01b25449e29d54fc4879d939db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399"
I20260928 10:44:11.347249 11260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ccf715075742f98e7cac7a818cb26b" candidate_uuid: "4c478f01b25449e29d54fc4879d939db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b2baa0c0424c25a3fee454e2c121f0"
I20260928 10:44:11.347457 11260 raft_consensus.cc:3037] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.347476 10980 raft_consensus.cc:3037] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.349639 10980 raft_consensus.cc:2454] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c478f01b25449e29d54fc4879d939db in term 1.
I20260928 10:44:11.349638 11260 raft_consensus.cc:2454] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c478f01b25449e29d54fc4879d939db in term 1.
I20260928 10:44:11.350207 11052 leader_election.cc:302] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c478f01b25449e29d54fc4879d939db, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:11.350556 11337 raft_consensus.cc:2781] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:11.351018 11337 raft_consensus.cc:686] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [term 1 LEADER]: Becoming Leader. State: Replica: 4c478f01b25449e29d54fc4879d939db, State: Running, Role: LEADER
I20260928 10:44:11.351526 11337 consensus_queue.cc:237] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.355914 11354 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: TxnStatusTablet state changed. Reason: New leader 4c478f01b25449e29d54fc4879d939db. Latest consensus state: current_term: 1 leader_uuid: "4c478f01b25449e29d54fc4879d939db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.356227 11354 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:44:11.356902 11356 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:44:11.356784 10823 catalog_manager.cc:5495] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c478f01b25449e29d54fc4879d939db (127.8.218.194). New cstate: current_term: 1 leader_uuid: "4c478f01b25449e29d54fc4879d939db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } health_report { overall_health: UNKNOWN } } }
I20260928 10:44:11.357167 11356 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:44:11.388455 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.381198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35805 (local address 127.0.0.1:41912)
0928 10:44:11.381481 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:11.381504 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:11.381668 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:11.382229 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:11.382232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:11.382243 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:44:11.382413 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.382421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.383968 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.383974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.385074 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.385093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.386280 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.386286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.386535 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.386611 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:44:11.387915 (+ 1304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:44:11.387940 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:44:11.388010 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 10:44:11.388169 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 10:44:11.388511 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.381312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.8.218.194:35805)
0928 10:44:11.381560 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.381567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.381591 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:44:11.381918 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.381924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.382042 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.382219 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:11.382560 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.383778 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.385268 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.386066 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.386233 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.386821 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:11.387763 (+ 942us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:11.387934 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.388058 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.388157 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 10:44:11.397572 11260 raft_consensus.cc:1261] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Refusing update from remote peer 4c478f01b25449e29d54fc4879d939db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:11.397670 10980 raft_consensus.cc:1261] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Refusing update from remote peer 4c478f01b25449e29d54fc4879d939db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:11.398877 11337 consensus_queue.cc:1035] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [LEADER]: Connected to new peer: Peer: permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:11.399430 11354 consensus_queue.cc:1035] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:11.412348 11336 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: TxnStatusTablet state changed. Reason: New leader 4c478f01b25449e29d54fc4879d939db. Latest consensus state: current_term: 1 leader_uuid: "4c478f01b25449e29d54fc4879d939db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.412678 11336 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:11.416683 11354 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c478f01b25449e29d54fc4879d939db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.417006 11354 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:44:11.422730 11336 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c478f01b25449e29d54fc4879d939db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.423069 11336 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P 29731b7a843a49e9a9e0a86f0664d399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:11.428589 11335 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: TxnStatusTablet state changed. Reason: New leader 4c478f01b25449e29d54fc4879d939db. Latest consensus state: current_term: 1 leader_uuid: "4c478f01b25449e29d54fc4879d939db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.428932 11335 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:11.435379 11337 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c478f01b25449e29d54fc4879d939db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.435745 11337 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P 4c478f01b25449e29d54fc4879d939db: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:44:11.438112 11361 mvcc.cc:205] Tried to move back new op lower bound from 7334265861712859136 to 7334265861539905536. Current Snapshot: MvccSnapshot[applied={T|T < 7334265861712859136}]
I20260928 10:44:11.447358 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.429644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.8.218.195:39885)
0928 10:44:11.429868 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.429875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.434309 (+ 4434us) server_negotiation.cc:407] Connection header received
0928 10:44:11.434592 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.434598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.434720 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.434901 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:11.439841 (+ 4940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.441104 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.443917 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.444657 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.444774 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.445613 (+ 839us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:11.446493 (+ 880us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:11.446648 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.446924 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.447030 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 10:44:11.450133 11335 tablet_replica.cc:439] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c478f01b25449e29d54fc4879d939db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } } }
I20260928 10:44:11.450508 11335 tablet_replica.cc:442] T 10ccf715075742f98e7cac7a818cb26b P e0b2baa0c0424c25a3fee454e2c121f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:44:11.479998 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.461607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:35805 (local address 127.8.218.195:43329)
0928 10:44:11.472162 (+ 10555us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:11.472195 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:11.472408 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:11.472857 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:11.472861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:11.472873 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:44:11.473248 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.473263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.475396 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.475401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.476733 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.476756 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.478035 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.478051 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.478208 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.478280 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:44:11.479401 (+ 1121us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:44:11.479416 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 10:44:11.479516 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 10:44:11.479670 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10311,"thread_start_us":10488,"threads_started":1}
I20260928 10:44:11.480042 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.461639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:43329 (local address 127.8.218.194:35805)
0928 10:44:11.461865 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.461872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.472340 (+ 10468us) server_negotiation.cc:407] Connection header received
0928 10:44:11.472570 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.472576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.472682 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.472809 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:11.473475 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.475060 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.476959 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.477688 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.478017 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.478612 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:11.478902 (+ 290us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:11.479419 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.479616 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.479722 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:44:11.494729 10980 raft_consensus.cc:1261] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Refusing update from remote peer e0b2baa0c0424c25a3fee454e2c121f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:11.495246 11120 raft_consensus.cc:1261] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Refusing update from remote peer e0b2baa0c0424c25a3fee454e2c121f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:11.495651 11335 consensus_queue.cc:1035] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:11.496366 11339 consensus_queue.cc:1035] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:11.505600 11360 mvcc.cc:205] Tried to move back new op lower bound from 7334265862113566720 to 7334265860237434880. Current Snapshot: MvccSnapshot[applied={T|T < 7334265862113566720}]
I20260928 10:44:11.517948 11359 mvcc.cc:205] Tried to move back new op lower bound from 7334265862113566720 to 7334265860237434880. Current Snapshot: MvccSnapshot[applied={T|T < 7334265862113566720}]
I20260928 10:44:11.547151 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.540142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:48321 (local address 127.8.218.195:39885)
0928 10:44:11.540316 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:11.540321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:11.541128 (+ 807us) server_negotiation.cc:407] Connection header received
0928 10:44:11.541468 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:11.541474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:11.541563 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:11.541614 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:11.542377 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.543467 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.545176 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.545799 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.545917 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.546382 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:11.546561 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:11.546646 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:11.546769 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 10:44:11.546851 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:44:11.547302 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:11.540225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:39885 (local address 127.8.218.194:48321)
0928 10:44:11.541010 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:11.541032 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:11.541251 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:11.541785 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:11.541788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:11.541798 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:44:11.542187 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.542201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.543643 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.543647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.545035 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:11.545049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:11.545990 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:11.545996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:11.546119 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:11.546199 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:44:11.546648 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:44:11.546660 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:44:11.546737 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 10:44:11.546890 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":352,"threads_started":1}
I20260928 10:44:11.605923 10823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:11.606267 10823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:11.613096 10823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: Sending DeleteTablet for 3 replicas of tablet 33faebecfb5a419da9a313eb8e520c2c
I20260928 10:44:11.615808 10960 tablet_service.cc:1514] Processing DeleteTablet for tablet 33faebecfb5a419da9a313eb8e520c2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:11 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:44234
I20260928 10:44:11.619050 11100 tablet_service.cc:1514] Processing DeleteTablet for tablet 33faebecfb5a419da9a313eb8e520c2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:11 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:41900
I20260928 10:44:11.619689 11240 tablet_service.cc:1514] Processing DeleteTablet for tablet 33faebecfb5a419da9a313eb8e520c2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:11 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:54490
I20260928 10:44:11.629390 10821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:11.630357 10821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:11.640862 10959 tablet_service.cc:1467] Processing CreateTablet for tablet 90829065470948bf8030ed6673a78453 (DEFAULT_TABLE table=test-table [id=8cbf42180e5c40ac9a64298ecaa10f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:11.641532 10959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90829065470948bf8030ed6673a78453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:11.641754 11099 tablet_service.cc:1467] Processing CreateTablet for tablet 90829065470948bf8030ed6673a78453 (DEFAULT_TABLE table=test-table [id=8cbf42180e5c40ac9a64298ecaa10f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:11.642362 11099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90829065470948bf8030ed6673a78453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:11.646479 11239 tablet_service.cc:1467] Processing CreateTablet for tablet 90829065470948bf8030ed6673a78453 (DEFAULT_TABLE table=test-table [id=8cbf42180e5c40ac9a64298ecaa10f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:11.647085 11239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90829065470948bf8030ed6673a78453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:11.648698 11331 tablet_bootstrap.cc:492] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: Bootstrap starting.
I20260928 10:44:11.650549 11331 tablet_bootstrap.cc:654] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:11.668452 11331 tablet_bootstrap.cc:492] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: No bootstrap required, opened a new log
I20260928 10:44:11.668692 11331 ts_tablet_manager.cc:1397] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.009s
I20260928 10:44:11.669623 11331 raft_consensus.cc:348] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.670084 11331 raft_consensus.cc:374] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:11.670235 11331 raft_consensus.cc:729] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29731b7a843a49e9a9e0a86f0664d399, State: Initialized, Role: FOLLOWER
I20260928 10:44:11.670678 11331 consensus_queue.cc:260] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.672657 11331 ts_tablet_manager.cc:1428] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:11.673545 11330 tablet_bootstrap.cc:492] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: Bootstrap starting.
I20260928 10:44:11.675655 11330 tablet_bootstrap.cc:654] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:11.687979 11388 tablet_bootstrap.cc:492] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: Bootstrap starting.
I20260928 10:44:11.694777 11388 tablet_bootstrap.cc:654] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:11.696134 11385 tablet_replica.cc:330] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: stopping tablet replica
I20260928 10:44:11.696427 11330 tablet_bootstrap.cc:492] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: No bootstrap required, opened a new log
I20260928 10:44:11.696727 11330 ts_tablet_manager.cc:1397] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.008s
I20260928 10:44:11.697464 11383 tablet_replica.cc:330] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: stopping tablet replica
I20260928 10:44:11.698103 11330 raft_consensus.cc:348] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.698611 11330 raft_consensus.cc:374] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:11.698750 11330 raft_consensus.cc:729] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Initialized, Role: FOLLOWER
I20260928 10:44:11.698714 11383 raft_consensus.cc:2229] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:11.699147 11383 raft_consensus.cc:2258] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:11.699191 11330 consensus_queue.cc:260] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.701061 11330 ts_tablet_manager.cc:1428] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 10:44:11.700917 11384 tablet_replica.cc:330] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: stopping tablet replica
I20260928 10:44:11.701838 11384 raft_consensus.cc:2229] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:11.702690 11384 raft_consensus.cc:2258] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:11.696695 11385 raft_consensus.cc:2229] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:11.705735 11383 ts_tablet_manager.cc:1905] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:11.707800 11385 raft_consensus.cc:2258] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:11.711180 11385 ts_tablet_manager.cc:1905] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:11.716730 11384 ts_tablet_manager.cc:1905] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:11.723954 11383 ts_tablet_manager.cc:1918] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:11.724189 11383 log.cc:1196] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/33faebecfb5a419da9a313eb8e520c2c
I20260928 10:44:11.724942 11383 ts_tablet_manager.cc:1939] T 33faebecfb5a419da9a313eb8e520c2c P 29731b7a843a49e9a9e0a86f0664d399: Deleting consensus metadata
I20260928 10:44:11.725229 11385 ts_tablet_manager.cc:1918] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:11.725566 11385 log.cc:1196] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/33faebecfb5a419da9a313eb8e520c2c
I20260928 10:44:11.726197 11385 ts_tablet_manager.cc:1939] T 33faebecfb5a419da9a313eb8e520c2c P e0b2baa0c0424c25a3fee454e2c121f0: Deleting consensus metadata
I20260928 10:44:11.726691 11388 tablet_bootstrap.cc:492] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: No bootstrap required, opened a new log
I20260928 10:44:11.726994 11388 ts_tablet_manager.cc:1397] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.008s
I20260928 10:44:11.727901 11388 raft_consensus.cc:348] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.728302 11388 raft_consensus.cc:374] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:11.728423 11388 raft_consensus.cc:729] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c478f01b25449e29d54fc4879d939db, State: Initialized, Role: FOLLOWER
I20260928 10:44:11.728776 11388 consensus_queue.cc:260] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.730634 11388 ts_tablet_manager.cc:1428] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260928 10:44:11.728420 11384 ts_tablet_manager.cc:1918] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:11.731120 11384 log.cc:1196] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/33faebecfb5a419da9a313eb8e520c2c
I20260928 10:44:11.731818 11384 ts_tablet_manager.cc:1939] T 33faebecfb5a419da9a313eb8e520c2c P 4c478f01b25449e29d54fc4879d939db: Deleting consensus metadata
I20260928 10:44:11.733017 10807 catalog_manager.cc:4841] TS 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153): tablet 33faebecfb5a419da9a313eb8e520c2c (table test-table [id=da78d09197424f9a8d1349373845fc70]) successfully deleted
I20260928 10:44:11.733752 10807 catalog_manager.cc:4841] TS 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805): tablet 33faebecfb5a419da9a313eb8e520c2c (table test-table [id=da78d09197424f9a8d1349373845fc70]) successfully deleted
I20260928 10:44:11.734863 10809 catalog_manager.cc:4841] TS e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195:39885): tablet 33faebecfb5a419da9a313eb8e520c2c (table test-table [id=da78d09197424f9a8d1349373845fc70]) successfully deleted
I20260928 10:44:11.838698 11379 raft_consensus.cc:482] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:11.838991 11379 raft_consensus.cc:504] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.840015 11379 leader_election.cc:288] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:11.840787 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90829065470948bf8030ed6673a78453" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399" is_pre_election: true
I20260928 10:44:11.841063 10980 raft_consensus.cc:2454] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 0.
I20260928 10:44:11.840969 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90829065470948bf8030ed6673a78453" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c478f01b25449e29d54fc4879d939db" is_pre_election: true
I20260928 10:44:11.841291 11119 raft_consensus.cc:2454] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 0.
I20260928 10:44:11.841698 11190 leader_election.cc:302] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29731b7a843a49e9a9e0a86f0664d399, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:11.842034 11379 raft_consensus.cc:2781] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:11.842150 11379 raft_consensus.cc:482] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:11.842248 11379 raft_consensus.cc:3037] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.845592 11379 raft_consensus.cc:504] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.846318 11379 leader_election.cc:288] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 election: Requested vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:11.847007 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90829065470948bf8030ed6673a78453" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399"
I20260928 10:44:11.847023 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90829065470948bf8030ed6673a78453" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c478f01b25449e29d54fc4879d939db"
I20260928 10:44:11.847246 10980 raft_consensus.cc:3037] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.847265 11119 raft_consensus.cc:3037] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:11.850284 10980 raft_consensus.cc:2454] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 1.
I20260928 10:44:11.850502 11119 raft_consensus.cc:2454] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 1.
I20260928 10:44:11.850824 11190 leader_election.cc:302] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29731b7a843a49e9a9e0a86f0664d399, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:11.851308 11379 raft_consensus.cc:2781] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:11.851681 11379 raft_consensus.cc:686] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 LEADER]: Becoming Leader. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Running, Role: LEADER
I20260928 10:44:11.852077 11379 consensus_queue.cc:237] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:11.855469 10821 catalog_manager.cc:5495] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } health_report { overall_health: HEALTHY } } }
I20260928 10:44:11.935494 11119 raft_consensus.cc:1261] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Refusing update from remote peer e0b2baa0c0424c25a3fee454e2c121f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:11.935505 10979 raft_consensus.cc:1261] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Refusing update from remote peer e0b2baa0c0424c25a3fee454e2c121f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:11.936365 11335 consensus_queue.cc:1035] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:11.936655 11339 consensus_queue.cc:1035] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:44:11.963843 11361 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 10:44:13.580140 10821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:13.580458 10821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:13.584960 10821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: Sending DeleteTablet for 3 replicas of tablet 90829065470948bf8030ed6673a78453
I20260928 10:44:13.586102 11100 tablet_service.cc:1514] Processing DeleteTablet for tablet 90829065470948bf8030ed6673a78453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:13 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:41900
I20260928 10:44:13.586166 10960 tablet_service.cc:1514] Processing DeleteTablet for tablet 90829065470948bf8030ed6673a78453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:13 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:44234
I20260928 10:44:13.586660 11240 tablet_service.cc:1514] Processing DeleteTablet for tablet 90829065470948bf8030ed6673a78453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:13 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:54490
I20260928 10:44:13.586834 11414 tablet_replica.cc:330] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: stopping tablet replica
I20260928 10:44:13.587023 11414 raft_consensus.cc:2229] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:13.587260 11414 raft_consensus.cc:2258] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:13.587710 11413 tablet_replica.cc:330] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: stopping tablet replica
I20260928 10:44:13.587872 11413 raft_consensus.cc:2229] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:13.588006 11415 tablet_replica.cc:330] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: stopping tablet replica
I20260928 10:44:13.588050 11413 raft_consensus.cc:2258] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:13.588564 11415 raft_consensus.cc:2229] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:13.588941 11415 raft_consensus.cc:2258] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:13.588967 11413 ts_tablet_manager.cc:1905] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:13.589927 11415 ts_tablet_manager.cc:1905] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:13.595222 11413 ts_tablet_manager.cc:1918] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:44:13.595355 11413 log.cc:1196] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/90829065470948bf8030ed6673a78453
I20260928 10:44:13.595484 11414 ts_tablet_manager.cc:1905] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:13.595836 11413 ts_tablet_manager.cc:1939] T 90829065470948bf8030ed6673a78453 P 4c478f01b25449e29d54fc4879d939db: Deleting consensus metadata
I20260928 10:44:13.596344 11415 ts_tablet_manager.cc:1918] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:44:13.596459 11415 log.cc:1196] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/90829065470948bf8030ed6673a78453
I20260928 10:44:13.596837 11415 ts_tablet_manager.cc:1939] T 90829065470948bf8030ed6673a78453 P e0b2baa0c0424c25a3fee454e2c121f0: Deleting consensus metadata
I20260928 10:44:13.596912 10807 catalog_manager.cc:4841] TS 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805): tablet 90829065470948bf8030ed6673a78453 (table test-table [id=8cbf42180e5c40ac9a64298ecaa10f8d]) successfully deleted
I20260928 10:44:13.597968 10809 catalog_manager.cc:4841] TS e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195:39885): tablet 90829065470948bf8030ed6673a78453 (table test-table [id=8cbf42180e5c40ac9a64298ecaa10f8d]) successfully deleted
I20260928 10:44:13.601213 11414 ts_tablet_manager.cc:1918] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:44:13.601330 11414 log.cc:1196] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/90829065470948bf8030ed6673a78453
I20260928 10:44:13.601707 11414 ts_tablet_manager.cc:1939] T 90829065470948bf8030ed6673a78453 P 29731b7a843a49e9a9e0a86f0664d399: Deleting consensus metadata
I20260928 10:44:13.602933 10807 catalog_manager.cc:4841] TS 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153): tablet 90829065470948bf8030ed6673a78453 (table test-table [id=8cbf42180e5c40ac9a64298ecaa10f8d]) successfully deleted
Sep 28 10:44:13 dist-test-slave-mn38 krb5kdc[10778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:44:13 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:13.647017 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:13.623280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.8.218.254:35751)
0928 10:44:13.623871 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:13.623877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:13.623902 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:44:13.624086 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:13.624090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:13.624171 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:13.624263 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:13.625654 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:13.626438 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:13.627764 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:13.628215 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:13.628334 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:13.641660 (+ 13326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:13.641666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:13.641670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:13.641704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:13.644250 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:13.645047 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:13.645052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:13.645054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:13.645243 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:13.645736 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:13.645739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:13.645741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:13.646013 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:13.646219 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:13.646564 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 10:44:13.646750 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":311,"threads_started":1}
I20260928 10:44:13.650636 10821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:13.651264 10821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:13.658099 11240 tablet_service.cc:1467] Processing CreateTablet for tablet 99d406a4970e46f1a161b280e76eb8d0 (DEFAULT_TABLE table=test-table [id=67a8ce17dfc44fc28dcb6bde3e043cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:13.658000 10960 tablet_service.cc:1467] Processing CreateTablet for tablet 99d406a4970e46f1a161b280e76eb8d0 (DEFAULT_TABLE table=test-table [id=67a8ce17dfc44fc28dcb6bde3e043cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:13.658072 11100 tablet_service.cc:1467] Processing CreateTablet for tablet 99d406a4970e46f1a161b280e76eb8d0 (DEFAULT_TABLE table=test-table [id=67a8ce17dfc44fc28dcb6bde3e043cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:13.658496 11100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d406a4970e46f1a161b280e76eb8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:13.658506 11240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d406a4970e46f1a161b280e76eb8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:13.658506 10960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d406a4970e46f1a161b280e76eb8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:13.665021 11424 tablet_bootstrap.cc:492] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: Bootstrap starting.
I20260928 10:44:13.665745 11425 tablet_bootstrap.cc:492] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: Bootstrap starting.
I20260928 10:44:13.666579 11426 tablet_bootstrap.cc:492] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: Bootstrap starting.
I20260928 10:44:13.667181 11424 tablet_bootstrap.cc:654] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:13.667735 11425 tablet_bootstrap.cc:654] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:13.668164 11426 tablet_bootstrap.cc:654] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:13.669986 11425 tablet_bootstrap.cc:492] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: No bootstrap required, opened a new log
I20260928 10:44:13.670166 11425 ts_tablet_manager.cc:1397] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 10:44:13.670487 11424 tablet_bootstrap.cc:492] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: No bootstrap required, opened a new log
I20260928 10:44:13.670634 11424 ts_tablet_manager.cc:1397] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 10:44:13.670856 11426 tablet_bootstrap.cc:492] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: No bootstrap required, opened a new log
I20260928 10:44:13.670993 11426 ts_tablet_manager.cc:1397] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 10:44:13.671042 11425 raft_consensus.cc:348] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.671296 11425 raft_consensus.cc:374] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:13.671401 11425 raft_consensus.cc:729] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29731b7a843a49e9a9e0a86f0664d399, State: Initialized, Role: FOLLOWER
I20260928 10:44:13.671408 11424 raft_consensus.cc:348] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.671666 11424 raft_consensus.cc:374] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:13.671653 11425 consensus_queue.cc:260] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.671859 11424 raft_consensus.cc:729] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c478f01b25449e29d54fc4879d939db, State: Initialized, Role: FOLLOWER
I20260928 10:44:13.671746 11426 raft_consensus.cc:348] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.672024 11426 raft_consensus.cc:374] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:13.672165 11426 raft_consensus.cc:729] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Initialized, Role: FOLLOWER
I20260928 10:44:13.672435 11426 consensus_queue.cc:260] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.673066 11426 ts_tablet_manager.cc:1428] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:44:13.674306 11425 ts_tablet_manager.cc:1428] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 10:44:13.672227 11424 consensus_queue.cc:260] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.676698 11424 ts_tablet_manager.cc:1428] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260928 10:44:13.952566 11379 raft_consensus.cc:482] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:13.952818 11379 raft_consensus.cc:504] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.953560 11379 leader_election.cc:288] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:13.954239 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d406a4970e46f1a161b280e76eb8d0" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399" is_pre_election: true
I20260928 10:44:13.954300 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d406a4970e46f1a161b280e76eb8d0" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c478f01b25449e29d54fc4879d939db" is_pre_election: true
I20260928 10:44:13.954505 11119 raft_consensus.cc:2454] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 0.
I20260928 10:44:13.954557 10980 raft_consensus.cc:2454] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 0.
I20260928 10:44:13.955040 11190 leader_election.cc:302] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c478f01b25449e29d54fc4879d939db, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:13.955327 11379 raft_consensus.cc:2781] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:13.955431 11379 raft_consensus.cc:482] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:13.955541 11379 raft_consensus.cc:3037] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:13.957491 11379 raft_consensus.cc:504] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.958091 11379 leader_election.cc:288] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 election: Requested vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:13.958781 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d406a4970e46f1a161b280e76eb8d0" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c478f01b25449e29d54fc4879d939db"
I20260928 10:44:13.958774 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d406a4970e46f1a161b280e76eb8d0" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399"
I20260928 10:44:13.958981 11119 raft_consensus.cc:3037] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:13.959048 10980 raft_consensus.cc:3037] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:13.961051 11119 raft_consensus.cc:2454] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 1.
I20260928 10:44:13.961090 10980 raft_consensus.cc:2454] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 1.
I20260928 10:44:13.961531 11190 leader_election.cc:302] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29731b7a843a49e9a9e0a86f0664d399, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:13.961861 11379 raft_consensus.cc:2781] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:13.962038 11379 raft_consensus.cc:686] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 LEADER]: Becoming Leader. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Running, Role: LEADER
I20260928 10:44:13.962347 11379 consensus_queue.cc:237] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:13.965333 10821 catalog_manager.cc:5495] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } health_report { overall_health: HEALTHY } } }
I20260928 10:44:14.048727 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:14.043674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.8.218.195:39885)
0928 10:44:14.044276 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:14.044280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:14.044297 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:44:14.044346 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:14.044349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:14.044407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:14.044488 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:14.044909 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:14.045729 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:14.046982 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:14.047404 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:14.047510 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:14.047980 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:14.048106 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:14.048180 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:14.048322 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 10:44:14.048400 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":270,"threads_started":1}
I20260928 10:44:14.058681 10980 raft_consensus.cc:1261] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Refusing update from remote peer e0b2baa0c0424c25a3fee454e2c121f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:14.058694 11119 raft_consensus.cc:1261] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Refusing update from remote peer e0b2baa0c0424c25a3fee454e2c121f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:14.059335 11404 consensus_queue.cc:1035] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:14.059672 11379 consensus_queue.cc:1035] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:14.107872 10821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:14.108140 10821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:14.111423 10821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: Sending DeleteTablet for 3 replicas of tablet 99d406a4970e46f1a161b280e76eb8d0
I20260928 10:44:14.112440 11100 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d406a4970e46f1a161b280e76eb8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:14 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:41900
I20260928 10:44:14.112440 10960 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d406a4970e46f1a161b280e76eb8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:14 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:44234
I20260928 10:44:14.112547 11240 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d406a4970e46f1a161b280e76eb8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:14 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:54490
I20260928 10:44:14.113838 10821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:14.114619 10821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:14.116369 11443 tablet_replica.cc:330] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: stopping tablet replica
I20260928 10:44:14.116636 11443 raft_consensus.cc:2229] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:14.116837 11441 tablet_replica.cc:330] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: stopping tablet replica
I20260928 10:44:14.117038 11442 tablet_replica.cc:330] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: stopping tablet replica
I20260928 10:44:14.117064 11443 raft_consensus.cc:2258] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:14.117285 11441 raft_consensus.cc:2229] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:14.117543 11441 raft_consensus.cc:2258] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:14.117746 11442 raft_consensus.cc:2229] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:14.117964 11442 raft_consensus.cc:2258] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:14.118168 11443 ts_tablet_manager.cc:1905] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:14.119338 11441 ts_tablet_manager.cc:1905] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:14.121279 11442 ts_tablet_manager.cc:1905] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:14.124452 11240 tablet_service.cc:1467] Processing CreateTablet for tablet 106b29d77b2f469eb8850a8db2cb03fe (DEFAULT_TABLE table=test-table [id=1812abe5fbf54dcf8a7cc8658d09b076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:14.124927 11240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106b29d77b2f469eb8850a8db2cb03fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:14.125249 11443 ts_tablet_manager.cc:1918] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:14.125388 11443 log.cc:1196] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/99d406a4970e46f1a161b280e76eb8d0
I20260928 10:44:14.125901 11443 ts_tablet_manager.cc:1939] T 99d406a4970e46f1a161b280e76eb8d0 P e0b2baa0c0424c25a3fee454e2c121f0: Deleting consensus metadata
I20260928 10:44:14.126669 11441 ts_tablet_manager.cc:1918] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:14.126801 11441 log.cc:1196] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/99d406a4970e46f1a161b280e76eb8d0
I20260928 10:44:14.127099 10809 catalog_manager.cc:4841] TS e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195:39885): tablet 99d406a4970e46f1a161b280e76eb8d0 (table test-table [id=67a8ce17dfc44fc28dcb6bde3e043cf9]) successfully deleted
I20260928 10:44:14.127214 11441 ts_tablet_manager.cc:1939] T 99d406a4970e46f1a161b280e76eb8d0 P 29731b7a843a49e9a9e0a86f0664d399: Deleting consensus metadata
I20260928 10:44:14.128307 10807 catalog_manager.cc:4841] TS 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153): tablet 99d406a4970e46f1a161b280e76eb8d0 (table test-table [id=67a8ce17dfc44fc28dcb6bde3e043cf9]) successfully deleted
I20260928 10:44:14.129096 10960 tablet_service.cc:1467] Processing CreateTablet for tablet 106b29d77b2f469eb8850a8db2cb03fe (DEFAULT_TABLE table=test-table [id=1812abe5fbf54dcf8a7cc8658d09b076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:14.129572 10960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106b29d77b2f469eb8850a8db2cb03fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:14.129869 11100 tablet_service.cc:1467] Processing CreateTablet for tablet 106b29d77b2f469eb8850a8db2cb03fe (DEFAULT_TABLE table=test-table [id=1812abe5fbf54dcf8a7cc8658d09b076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:14.130261 11100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106b29d77b2f469eb8850a8db2cb03fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:14.130570 11442 ts_tablet_manager.cc:1918] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:44:14.130707 11442 log.cc:1196] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/99d406a4970e46f1a161b280e76eb8d0
I20260928 10:44:14.131197 11442 ts_tablet_manager.cc:1939] T 99d406a4970e46f1a161b280e76eb8d0 P 4c478f01b25449e29d54fc4879d939db: Deleting consensus metadata
I20260928 10:44:14.132364 10807 catalog_manager.cc:4841] TS 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805): tablet 99d406a4970e46f1a161b280e76eb8d0 (table test-table [id=67a8ce17dfc44fc28dcb6bde3e043cf9]) successfully deleted
I20260928 10:44:14.135391 11424 tablet_bootstrap.cc:492] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: Bootstrap starting.
I20260928 10:44:14.135790 11426 tablet_bootstrap.cc:492] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: Bootstrap starting.
I20260928 10:44:14.137065 11424 tablet_bootstrap.cc:654] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:14.137476 11426 tablet_bootstrap.cc:654] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:14.137514 11425 tablet_bootstrap.cc:492] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: Bootstrap starting.
I20260928 10:44:14.139168 11425 tablet_bootstrap.cc:654] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:14.139283 11424 tablet_bootstrap.cc:492] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: No bootstrap required, opened a new log
I20260928 10:44:14.139451 11424 ts_tablet_manager.cc:1397] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:14.139654 11426 tablet_bootstrap.cc:492] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: No bootstrap required, opened a new log
I20260928 10:44:14.139780 11426 ts_tablet_manager.cc:1397] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 10:44:14.140308 11424 raft_consensus.cc:348] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.140518 11424 raft_consensus.cc:374] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:14.140413 11426 raft_consensus.cc:348] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.140631 11424 raft_consensus.cc:729] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c478f01b25449e29d54fc4879d939db, State: Initialized, Role: FOLLOWER
I20260928 10:44:14.140681 11426 raft_consensus.cc:374] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:14.140793 11426 raft_consensus.cc:729] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Initialized, Role: FOLLOWER
I20260928 10:44:14.140928 11424 consensus_queue.cc:260] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.141014 11426 consensus_queue.cc:260] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.141155 11425 tablet_bootstrap.cc:492] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: No bootstrap required, opened a new log
I20260928 10:44:14.141310 11425 ts_tablet_manager.cc:1397] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:14.141671 11426 ts_tablet_manager.cc:1428] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 10:44:14.141690 11424 ts_tablet_manager.cc:1428] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 10:44:14.142318 11425 raft_consensus.cc:348] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.142741 11425 raft_consensus.cc:374] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:14.142835 11425 raft_consensus.cc:729] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29731b7a843a49e9a9e0a86f0664d399, State: Initialized, Role: FOLLOWER
I20260928 10:44:14.143064 11425 consensus_queue.cc:260] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.143641 11425 ts_tablet_manager.cc:1428] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 10:44:14.189690 11404 raft_consensus.cc:482] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:14.189925 11404 raft_consensus.cc:504] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.190641 11404 leader_election.cc:288] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:14.191334 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b29d77b2f469eb8850a8db2cb03fe" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399" is_pre_election: true
I20260928 10:44:14.191368 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b29d77b2f469eb8850a8db2cb03fe" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c478f01b25449e29d54fc4879d939db" is_pre_election: true
I20260928 10:44:14.191617 10980 raft_consensus.cc:2454] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 0.
I20260928 10:44:14.191624 11119 raft_consensus.cc:2454] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 0.
I20260928 10:44:14.192137 11190 leader_election.cc:302] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29731b7a843a49e9a9e0a86f0664d399, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:14.192492 11404 raft_consensus.cc:2781] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:14.192601 11404 raft_consensus.cc:482] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:14.192719 11404 raft_consensus.cc:3037] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:14.194852 11404 raft_consensus.cc:504] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.195443 11404 leader_election.cc:288] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 election: Requested vote from peers 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153), 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805)
I20260928 10:44:14.196049 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b29d77b2f469eb8850a8db2cb03fe" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29731b7a843a49e9a9e0a86f0664d399"
I20260928 10:44:14.196112 11119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b29d77b2f469eb8850a8db2cb03fe" candidate_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c478f01b25449e29d54fc4879d939db"
I20260928 10:44:14.196261 10980 raft_consensus.cc:3037] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:14.196316 11119 raft_consensus.cc:3037] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:14.198572 11119 raft_consensus.cc:2454] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 1.
I20260928 10:44:14.198577 10980 raft_consensus.cc:2454] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b2baa0c0424c25a3fee454e2c121f0 in term 1.
I20260928 10:44:14.199082 11190 leader_election.cc:302] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c478f01b25449e29d54fc4879d939db, e0b2baa0c0424c25a3fee454e2c121f0; no voters:
I20260928 10:44:14.199462 11404 raft_consensus.cc:2781] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:14.199719 11404 raft_consensus.cc:686] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 LEADER]: Becoming Leader. State: Replica: e0b2baa0c0424c25a3fee454e2c121f0, State: Running, Role: LEADER
I20260928 10:44:14.200001 11404 consensus_queue.cc:237] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } }
I20260928 10:44:14.202658 10823 catalog_manager.cc:5495] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b2baa0c0424c25a3fee454e2c121f0" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39885 } health_report { overall_health: HEALTHY } } }
I20260928 10:44:14.230548 10980 raft_consensus.cc:1261] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Refusing update from remote peer e0b2baa0c0424c25a3fee454e2c121f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:14.230540 11119 raft_consensus.cc:1261] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Refusing update from remote peer e0b2baa0c0424c25a3fee454e2c121f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:14.231120 11404 consensus_queue.cc:1035] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29731b7a843a49e9a9e0a86f0664d399" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:14.231470 11438 consensus_queue.cc:1035] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c478f01b25449e29d54fc4879d939db" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:44:14.253334 11396 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 10:44:15.839861 10822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:15.840154 10822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:15.846262 10822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bce17920f39c47b2b1ea398334bf61c0: Sending DeleteTablet for 3 replicas of tablet 106b29d77b2f469eb8850a8db2cb03fe
I20260928 10:44:15.847208 11240 tablet_service.cc:1514] Processing DeleteTablet for tablet 106b29d77b2f469eb8850a8db2cb03fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:15 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:54490
I20260928 10:44:15.847205 10960 tablet_service.cc:1514] Processing DeleteTablet for tablet 106b29d77b2f469eb8850a8db2cb03fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:15 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:44234
I20260928 10:44:15.847414 11100 tablet_service.cc:1514] Processing DeleteTablet for tablet 106b29d77b2f469eb8850a8db2cb03fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:15 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:41900
I20260928 10:44:15.848140 11473 tablet_replica.cc:330] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: stopping tablet replica
I20260928 10:44:15.848356 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10894
I20260928 10:44:15.849257 11472 tablet_replica.cc:330] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: stopping tablet replica
I20260928 10:44:15.849563 11472 raft_consensus.cc:2229] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:15.849936 11473 raft_consensus.cc:2229] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:15.850486 11472 raft_consensus.cc:2258] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:15.850831 11473 raft_consensus.cc:2258] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:15.851020 11474 tablet_replica.cc:330] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: stopping tablet replica
I20260928 10:44:15.851246 11474 raft_consensus.cc:2229] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:15.851505 11474 raft_consensus.cc:2258] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:15.852447 11473 ts_tablet_manager.cc:1905] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:15.852577 11472 ts_tablet_manager.cc:1905] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:15.852732 11474 ts_tablet_manager.cc:1905] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:15.859977 11474 ts_tablet_manager.cc:1918] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:44:15.859978 11472 ts_tablet_manager.cc:1918] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:44:15.859977 11473 ts_tablet_manager.cc:1918] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 10:44:15.860148 11474 log.cc:1196] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/106b29d77b2f469eb8850a8db2cb03fe
I20260928 10:44:15.860147 11472 log.cc:1196] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/106b29d77b2f469eb8850a8db2cb03fe
I20260928 10:44:15.860354 11473 log.cc:1196] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/106b29d77b2f469eb8850a8db2cb03fe
I20260928 10:44:15.860606 11472 ts_tablet_manager.cc:1939] T 106b29d77b2f469eb8850a8db2cb03fe P e0b2baa0c0424c25a3fee454e2c121f0: Deleting consensus metadata
I20260928 10:44:15.860632 11474 ts_tablet_manager.cc:1939] T 106b29d77b2f469eb8850a8db2cb03fe P 4c478f01b25449e29d54fc4879d939db: Deleting consensus metadata
I20260928 10:44:15.860980 11473 ts_tablet_manager.cc:1939] T 106b29d77b2f469eb8850a8db2cb03fe P 29731b7a843a49e9a9e0a86f0664d399: Deleting consensus metadata
I20260928 10:44:15.861853 10807 catalog_manager.cc:4841] TS 4c478f01b25449e29d54fc4879d939db (127.8.218.194:35805): tablet 106b29d77b2f469eb8850a8db2cb03fe (table test-table [id=1812abe5fbf54dcf8a7cc8658d09b076]) successfully deleted
I20260928 10:44:15.861896 10809 catalog_manager.cc:4841] TS e0b2baa0c0424c25a3fee454e2c121f0 (127.8.218.195:39885): tablet 106b29d77b2f469eb8850a8db2cb03fe (table test-table [id=1812abe5fbf54dcf8a7cc8658d09b076]) successfully deleted
Sep 28 10:44:15 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:44:15.862744 10807 catalog_manager.cc:4841] TS 29731b7a843a49e9a9e0a86f0664d399 (127.8.218.193:34153): tablet 106b29d77b2f469eb8850a8db2cb03fe (table test-table [id=1812abe5fbf54dcf8a7cc8658d09b076]) successfully deleted
I20260928 10:44:15.869560 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:15.849648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.8.218.193:34153)
0928 10:44:15.851217 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:15.851493 (+ 276us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:15.851970 (+ 477us) server_negotiation.cc:407] Connection header received
0928 10:44:15.852175 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:15.852180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:15.852272 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:15.852384 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:15.854285 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:15.855432 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:15.857282 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:15.858033 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:15.858192 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:15.863568 (+ 5376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:15.863586 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:15.863604 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:15.863645 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:15.866582 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:15.867377 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:15.867387 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:15.867389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:15.867542 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:15.867931 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:15.867935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:15.867937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:15.868315 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:15.868620 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:15.868930 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 10:44:15.869168 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1422,"thread_start_us":1300,"threads_started":1}
W20260928 10:44:15.870195 11025 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54510
W20260928 10:44:15.870860 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10894: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:15.871052 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 10894
I20260928 10:44:15.886564 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11034
Sep 28 10:44:15 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:44:15.902313 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:15.887313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.8.218.194:35805)
0928 10:44:15.887821 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:15.887825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:15.887842 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:44:15.887894 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:15.887898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:15.887964 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:15.888087 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:15.889399 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:15.890198 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:15.891546 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:15.891958 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:15.892077 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:15.896620 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:15.896640 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:15.896658 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:15.896704 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:15.899527 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:15.900345 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:15.900353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:15.900355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:15.900532 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:15.900857 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:15.900860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:15.900862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:15.901221 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:15.901578 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:15.901845 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 10:44:15.902028 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
W20260928 10:44:15.903098 11164 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57666
W20260928 10:44:15.903712 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11034: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:15.903869 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 11034
I20260928 10:44:15.921988 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11174
Sep 28 10:44:15 dist-test-slave-mn38 krb5kdc[10778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592253, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:44:15.938485 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:15.922713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.8.218.195:39885)
0928 10:44:15.923226 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:15.923230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:15.923245 (+ 15us) server_negotiation.cc:407] Connection header received
0928 10:44:15.923297 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:15.923300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:15.923356 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:15.923449 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:15.924807 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:15.925740 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:15.927074 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:15.927482 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:15.927586 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:15.932241 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:15.932259 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:15.932278 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:15.932324 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:15.935570 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:15.936379 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:15.936387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:15.936390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:15.936531 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:15.936917 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:15.936922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:15.936924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:15.937295 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:15.937617 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:15.937902 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 10:44:15.938130 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
W20260928 10:44:15.939246 11305 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53568
W20260928 10:44:15.939925 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11174: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:15.940083 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 11174
I20260928 10:44:15.958119 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 10791
I20260928 10:44:15.959306 10863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:16.077585 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 10791
2026-09-28T10:44:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (12066 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:16 dist-test-slave-mn38 krb5kdc[11482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:16 dist-test-slave-mn38 krb5kdc[11482](info): set up 2 sockets
Sep 28 10:44:16 dist-test-slave-mn38 krb5kdc[11482](info): commencing operation
krb5kdc: starting...
W20260928 10:44:17.102517 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:44:17 dist-test-slave-mn38 krb5kdc[11482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:44:17.293237 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37263
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:17.622260 11495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:17.622553 11495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:17.622678 11495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:17.629931 11495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:17.630044 11495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:17.630107 11495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:17.630184 11495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:17.630244 11495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:17.640304 11495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:37263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:17.641709 11495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:17.643075 11495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:17.655515 11500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:17.655542 11503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:17.655555 11495 server_base.cc:1034] running on GCE node
W20260928 10:44:17.655531 11501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:17.656726 11495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:17.658306 11495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:17.659528 11495 hybrid_clock.cc:648] HybridClock initialized: now 1790592257659465 us; error 52 us; skew 500 ppm
I20260928 10:44:17.659868 11495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:17 dist-test-slave-mn38 krb5kdc[11482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592257, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:17.663040 11495 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:44:17.664587 11495 webserver.cc:460] Webserver started at http://127.8.218.254:32775/ using document root <none> and password file <none>
I20260928 10:44:17.665297 11495 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:17.665419 11495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:17.665736 11495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:17.668737 11495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "81b0945fa6c74dfab4d9c3a87d13de3d"
format_stamp: "Formatted at 2026-09-28 10:44:17 on dist-test-slave-mn38"
server_key: "7e5545cc0ec4dc2534e9574bfaeb4c29"
server_key_iv: "d3da0608dda3c05f63a8505ffc005594"
server_key_version: "encryptionkey@0"
I20260928 10:44:17.669555 11495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "81b0945fa6c74dfab4d9c3a87d13de3d"
format_stamp: "Formatted at 2026-09-28 10:44:17 on dist-test-slave-mn38"
server_key: "7e5545cc0ec4dc2534e9574bfaeb4c29"
server_key_iv: "d3da0608dda3c05f63a8505ffc005594"
server_key_version: "encryptionkey@0"
I20260928 10:44:17.674844 11495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:44:17.678476 11510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:17.679977 11495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:44:17.680169 11495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "81b0945fa6c74dfab4d9c3a87d13de3d"
format_stamp: "Formatted at 2026-09-28 10:44:17 on dist-test-slave-mn38"
server_key: "7e5545cc0ec4dc2534e9574bfaeb4c29"
server_key_iv: "d3da0608dda3c05f63a8505ffc005594"
server_key_version: "encryptionkey@0"
I20260928 10:44:17.680374 11495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:17.708312 11495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:17.712970 11495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:17.713240 11495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:17.731416 11495 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:37263
I20260928 10:44:17.731422 11561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:37263 every 8 connection(s)
I20260928 10:44:17.732872 11495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:17.736666 11562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:17.741825 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 11495
I20260928 10:44:17.742017 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:17.742599 9067 external_mini_cluster.cc:1428] Setting key 547f6fe624eef60f1ec37d61d0c16603
I20260928 10:44:17.747869 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d: Bootstrap starting.
I20260928 10:44:17.751713 11562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:17.752838 11562 log.cc:824] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:17.756173 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d: No bootstrap required, opened a new log
Sep 28 10:44:17 dist-test-slave-mn38 krb5kdc[11482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:17.766396 11562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b0945fa6c74dfab4d9c3a87d13de3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37263 } }
I20260928 10:44:17.766893 11562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:17.767007 11562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b0945fa6c74dfab4d9c3a87d13de3d, State: Initialized, Role: FOLLOWER
I20260928 10:44:17.767539 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:17.744789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.8.218.254:37263)
0928 10:44:17.745718 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:17.745736 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:17.745848 (+ 112us) server_negotiation.cc:407] Connection header received
0928 10:44:17.747563 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:17.747589 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:17.748056 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:17.748435 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:17.750347 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:17.751814 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:17.753137 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:17.754108 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:17.754257 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:17.758440 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:17.758450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:17.758472 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:17.758532 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:17.762735 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:17.763549 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:17.763558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:17.763568 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:17.763800 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:17.764133 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:17.764138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:17.764142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:17.764932 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:17.765223 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:17.765631 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 10:44:17.766091 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":689,"thread_start_us":408,"threads_started":1}
I20260928 10:44:17.767673 11562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b0945fa6c74dfab4d9c3a87d13de3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37263 } }
I20260928 10:44:17.767952 11562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:17.768095 11562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:17.768265 11562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:17.770884 11562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b0945fa6c74dfab4d9c3a87d13de3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37263 } }
I20260928 10:44:17.771445 11562 leader_election.cc:302] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81b0945fa6c74dfab4d9c3a87d13de3d; no voters:
I20260928 10:44:17.772079 11562 leader_election.cc:288] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:17.772228 11567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:17.772840 11567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [term 1 LEADER]: Becoming Leader. State: Replica: 81b0945fa6c74dfab4d9c3a87d13de3d, State: Running, Role: LEADER
I20260928 10:44:17.773418 11567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b0945fa6c74dfab4d9c3a87d13de3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37263 } }
I20260928 10:44:17.774075 11562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:17.776229 11568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81b0945fa6c74dfab4d9c3a87d13de3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b0945fa6c74dfab4d9c3a87d13de3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37263 } } }
I20260928 10:44:17.776645 11568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:17.776333 11569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81b0945fa6c74dfab4d9c3a87d13de3d. Latest consensus state: current_term: 1 leader_uuid: "81b0945fa6c74dfab4d9c3a87d13de3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b0945fa6c74dfab4d9c3a87d13de3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37263 } } }
I20260928 10:44:17.778203 11569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:17.779691 11574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:17.785290 11574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:17.796037 11574 catalog_manager.cc:1347] Generated new cluster ID: 7bc00aa0696e45baaf035cd4a029b368
I20260928 10:44:17.796172 11574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:17.809420 11574 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:17.811049 11574 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:17.821066 11574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81b0945fa6c74dfab4d9c3a87d13de3d: Generated new TSK 0
I20260928 10:44:17.822124 11574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:44:17.877568 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37263
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:18.237311 11588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:18.237668 11588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:18.237787 11588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:18.245096 11588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:18.245226 11588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:18.245342 11588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:18.255661 11588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:18.257038 11588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:18.258504 11588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:18.271770 11593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:18.271801 11594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:18.271988 11596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:18.272696 11588 server_base.cc:1034] running on GCE node
I20260928 10:44:18.273334 11588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:18.274425 11588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:18.275637 11588 hybrid_clock.cc:648] HybridClock initialized: now 1790592258275580 us; error 50 us; skew 500 ppm
I20260928 10:44:18.276007 11588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:18 dist-test-slave-mn38 krb5kdc[11482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592258, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:18.279284 11588 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:44:18.280887 11588 webserver.cc:460] Webserver started at http://127.8.218.193:38137/ using document root <none> and password file <none>
I20260928 10:44:18.281653 11588 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:18.281769 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:18.282068 11588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:18.285274 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "69b794a69bf34e089318b4a20b07ecfd"
format_stamp: "Formatted at 2026-09-28 10:44:18 on dist-test-slave-mn38"
server_key: "0bc22cc89c1b0a0058e690f254cadfdc"
server_key_iv: "472d5a2709cef148a73b052cde1bfbd1"
server_key_version: "encryptionkey@0"
I20260928 10:44:18.286154 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "69b794a69bf34e089318b4a20b07ecfd"
format_stamp: "Formatted at 2026-09-28 10:44:18 on dist-test-slave-mn38"
server_key: "0bc22cc89c1b0a0058e690f254cadfdc"
server_key_iv: "472d5a2709cef148a73b052cde1bfbd1"
server_key_version: "encryptionkey@0"
I20260928 10:44:18.291903 11588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:44:18.295910 11603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:18.297557 11588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:44:18.297734 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "69b794a69bf34e089318b4a20b07ecfd"
format_stamp: "Formatted at 2026-09-28 10:44:18 on dist-test-slave-mn38"
server_key: "0bc22cc89c1b0a0058e690f254cadfdc"
server_key_iv: "472d5a2709cef148a73b052cde1bfbd1"
server_key_version: "encryptionkey@0"
I20260928 10:44:18.297952 11588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:18.338011 11588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:18.342397 11588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:18.342680 11588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:18.343629 11588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:18.344970 11588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:18.345048 11588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:18.345166 11588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:18.345218 11588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:18.375620 11588 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:38475
I20260928 10:44:18.375674 11715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:38475 every 8 connection(s)
I20260928 10:44:18.376921 11588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:18.380173 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 11588
I20260928 10:44:18.380362 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:18.381171 9067 external_mini_cluster.cc:1428] Setting key 21e806e2b631202a72ccbad87ee0f5f6
Sep 28 10:44:18 dist-test-slave-mn38 krb5kdc[11482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592258, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:44:18.398411 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:18.379480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:33819 (local address 127.8.218.254:37263)
0928 10:44:18.379931 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:18.379936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:18.381098 (+ 1162us) server_negotiation.cc:407] Connection header received
0928 10:44:18.383138 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:18.383143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:18.383233 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:18.383322 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:18.385022 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:18.385868 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:18.387561 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:18.387999 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:18.388110 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:18.392299 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:18.392334 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:18.392338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:18.392376 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:18.394966 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:18.395985 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:18.395989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:18.395991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:18.396110 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:18.396575 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:18.396578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:18.396580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:18.396826 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:18.396932 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:18.397918 (+ 986us) server_negotiation.cc:299] Negotiation successful
0928 10:44:18.398095 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":240,"threads_started":1}
I20260928 10:44:18.399578 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:18.380096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37263 (local address 127.8.218.193:33819)
0928 10:44:18.380937 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:18.380982 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:18.382866 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:18.383573 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:18.383582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:18.384218 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:18.384843 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:18.384868 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:18.386034 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:18.386039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:18.387399 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:18.387416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:18.388233 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:18.388241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:18.388529 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:18.389238 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:18.389268 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:18.392034 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:18.395155 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:18.395166 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:18.395189 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:18.395849 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:18.396214 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:18.396219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:18.396223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:18.396479 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:18.396954 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:18.396963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:18.397466 (+ 503us) client_negotiation.cc:769] Sending connection context
0928 10:44:18.397841 (+ 375us) client_negotiation.cc:240] Negotiation successful
0928 10:44:18.398405 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
I20260928 10:44:18.401304 11716 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37263
I20260928 10:44:18.401685 11716 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:18.402467 11716 heartbeater.cc:511] Master 127.8.218.254:37263 requested a full tablet report, sending...
I20260928 10:44:18.404836 11527 ts_manager.cc:194] Registered new tserver with Master: 69b794a69bf34e089318b4a20b07ecfd (127.8.218.193:38475)
I20260928 10:44:18.406610 11527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:33819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:44:18.424170 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37263
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:18.756783 11722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:18.757107 11722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:18.757242 11722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:18.764887 11722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:18.765033 11722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:18.765184 11722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:18.775512 11722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:18.776901 11722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:18.778245 11722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:18.791644 11727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:18.791721 11728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:18.792335 11730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:18.793644 11722 server_base.cc:1034] running on GCE node
I20260928 10:44:18.794504 11722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:18.796769 11722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:18.797992 11722 hybrid_clock.cc:648] HybridClock initialized: now 1790592258797923 us; error 66 us; skew 500 ppm
I20260928 10:44:18.798339 11722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:18 dist-test-slave-mn38 krb5kdc[11482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592258, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:18.807587 11722 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:44:18.809410 11722 webserver.cc:460] Webserver started at http://127.8.218.194:43385/ using document root <none> and password file <none>
I20260928 10:44:18.810227 11722 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:18.810372 11722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:18.810716 11722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:18.813951 11722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "5939f4f4f0434fcfbf9895f28793dfbc"
format_stamp: "Formatted at 2026-09-28 10:44:18 on dist-test-slave-mn38"
server_key: "40c687b20d1f83de1210444f90c6d5b9"
server_key_iv: "1e1209421065bff36f1f90c162bd08a7"
server_key_version: "encryptionkey@0"
I20260928 10:44:18.814832 11722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "5939f4f4f0434fcfbf9895f28793dfbc"
format_stamp: "Formatted at 2026-09-28 10:44:18 on dist-test-slave-mn38"
server_key: "40c687b20d1f83de1210444f90c6d5b9"
server_key_iv: "1e1209421065bff36f1f90c162bd08a7"
server_key_version: "encryptionkey@0"
I20260928 10:44:18.820489 11722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 10:44:18.824287 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:18.825955 11722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:44:18.826124 11722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "5939f4f4f0434fcfbf9895f28793dfbc"
format_stamp: "Formatted at 2026-09-28 10:44:18 on dist-test-slave-mn38"
server_key: "40c687b20d1f83de1210444f90c6d5b9"
server_key_iv: "1e1209421065bff36f1f90c162bd08a7"
server_key_version: "encryptionkey@0"
I20260928 10:44:18.826327 11722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:18.854784 11722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:18.859107 11722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:18.859386 11722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:18.860355 11722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:18.861720 11722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:18.861799 11722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:18.861915 11722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:18.861967 11722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:18.892839 11722 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:36605
I20260928 10:44:18.892839 11849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:36605 every 8 connection(s)
I20260928 10:44:18.894214 11722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:18.903208 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 11722
I20260928 10:44:18.903352 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:18.903780 9067 external_mini_cluster.cc:1428] Setting key 6aecad982735a9f4383a6e65baecff93
Sep 28 10:44:18 dist-test-slave-mn38 krb5kdc[11482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592258, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:18.915050 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:18.896885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:52169 (local address 127.8.218.254:37263)
0928 10:44:18.897096 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:18.897102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:18.898360 (+ 1258us) server_negotiation.cc:407] Connection header received
0928 10:44:18.900174 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:18.900181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:18.900272 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:18.900378 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:18.902035 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:18.902766 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:18.904524 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:18.905023 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:18.905169 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:18.909076 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:18.909081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:18.909084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:18.909118 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:18.911943 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:18.912700 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:18.912706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:18.912710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:18.912873 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:18.913281 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:18.913286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:18.913289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:18.913650 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:18.913793 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:18.914483 (+ 690us) server_negotiation.cc:299] Negotiation successful
0928 10:44:18.914711 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":78}
I20260928 10:44:18.915971 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:18.897485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37263 (local address 127.8.218.194:52169)
0928 10:44:18.898188 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:18.898227 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:18.899871 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:18.900609 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:18.900619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:18.901152 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:18.901859 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:18.901882 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:18.902961 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:18.902965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:18.904352 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:18.904365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:18.905259 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:18.905266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:18.905582 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:18.906274 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:18.906296 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:18.908791 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:18.912094 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:18.912102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:18.912118 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:18.912571 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:18.912981 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:18.912984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:18.912986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:18.913173 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:18.913761 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:18.913768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:18.914125 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 10:44:18.914354 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 10:44:18.914769 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":254,"threads_started":1}
I20260928 10:44:18.917481 11850 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37263
I20260928 10:44:18.917852 11850 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:18.918620 11850 heartbeater.cc:511] Master 127.8.218.254:37263 requested a full tablet report, sending...
I20260928 10:44:18.920248 11527 ts_manager.cc:194] Registered new tserver with Master: 5939f4f4f0434fcfbf9895f28793dfbc (127.8.218.194:36605)
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:44:18.921216 11527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:52169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:44:18.948632 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37263
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:19.278236 11855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:19.278546 11855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:19.278666 11855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:19.286660 11855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:19.286796 11855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:19.286952 11855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:19.297821 11855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:19.299167 11855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:19.300499 11855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:19.313984 11861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:19.313956 11863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:19.313907 11860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:19.314952 11855 server_base.cc:1034] running on GCE node
I20260928 10:44:19.315634 11855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:19.316771 11855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:19.318013 11855 hybrid_clock.cc:648] HybridClock initialized: now 1790592259317956 us; error 53 us; skew 500 ppm
I20260928 10:44:19.318377 11855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:19 dist-test-slave-mn38 krb5kdc[11482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592259, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:19.321590 11855 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:44:19.323130 11855 webserver.cc:460] Webserver started at http://127.8.218.195:41049/ using document root <none> and password file <none>
I20260928 10:44:19.323915 11855 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:19.324020 11855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:19.324287 11855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:19.327353 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "f0aefd76b5e54007888649814e942cc2"
format_stamp: "Formatted at 2026-09-28 10:44:19 on dist-test-slave-mn38"
server_key: "bd49d85b778544558664475d771deb14"
server_key_iv: "1b7e7cf276318082c870f38cd13337d0"
server_key_version: "encryptionkey@0"
I20260928 10:44:19.328127 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "f0aefd76b5e54007888649814e942cc2"
format_stamp: "Formatted at 2026-09-28 10:44:19 on dist-test-slave-mn38"
server_key: "bd49d85b778544558664475d771deb14"
server_key_iv: "1b7e7cf276318082c870f38cd13337d0"
server_key_version: "encryptionkey@0"
I20260928 10:44:19.333777 11855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:44:19.337412 11870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:19.338961 11855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:44:19.339123 11855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "f0aefd76b5e54007888649814e942cc2"
format_stamp: "Formatted at 2026-09-28 10:44:19 on dist-test-slave-mn38"
server_key: "bd49d85b778544558664475d771deb14"
server_key_iv: "1b7e7cf276318082c870f38cd13337d0"
server_key_version: "encryptionkey@0"
I20260928 10:44:19.339318 11855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:19.354439 11855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:19.358907 11855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:19.359187 11855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:19.360167 11855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:19.361655 11855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:19.361734 11855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:19.361846 11855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:19.361900 11855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:19.395560 11855 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:38485
I20260928 10:44:19.395630 11982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:38485 every 8 connection(s)
I20260928 10:44:19.396821 11855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:19.407135 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 11855
I20260928 10:44:19.407327 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:19.407838 9067 external_mini_cluster.cc:1428] Setting key 9763f2715daf6e7fac4e6d775d37c13e
I20260928 10:44:19.410087 11716 heartbeater.cc:503] Master 127.8.218.254:37263 was elected leader, sending a full tablet report...
Sep 28 10:44:19 dist-test-slave-mn38 krb5kdc[11482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592259, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:19.417855 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:19.399388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:52765 (local address 127.8.218.254:37263)
0928 10:44:19.399560 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:19.399565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:19.400807 (+ 1242us) server_negotiation.cc:407] Connection header received
0928 10:44:19.402766 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:19.402770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:19.402882 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:19.402976 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:19.404749 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:19.405615 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:19.407623 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:19.408142 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:19.408257 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:19.412454 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:19.412459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:19.412462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:19.412495 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:19.415003 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:19.415744 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:19.415748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:19.415751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:19.415877 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:19.416251 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:19.416254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:19.416256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:19.416516 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:19.416621 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:19.417331 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 10:44:19.417554 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:44:19.418819 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:19.399854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37263 (local address 127.8.218.195:52765)
0928 10:44:19.400575 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:19.400613 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:19.402412 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:19.403249 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:19.403260 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:19.403862 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:19.404516 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:19.404537 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:19.405792 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:19.405796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:19.407456 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:19.407474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:19.408376 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:19.408382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:19.408631 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:19.409391 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:19.409426 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:19.412247 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:19.415144 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:19.415153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:19.415168 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:19.415603 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:19.415964 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:19.415967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:19.415969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:19.416157 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:19.416608 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:19.416614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:19.416973 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 10:44:19.417198 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 10:44:19.417668 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":282,"threads_started":1}
I20260928 10:44:19.420186 11983 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37263
I20260928 10:44:19.420463 11983 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:19.421178 11983 heartbeater.cc:511] Master 127.8.218.254:37263 requested a full tablet report, sending...
I20260928 10:44:19.422652 11527 ts_manager.cc:194] Registered new tserver with Master: f0aefd76b5e54007888649814e942cc2 (127.8.218.195:38485)
I20260928 10:44:19.423610 11527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:52765
I20260928 10:44:19.423771 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 10:44:19.453364 11718 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:19.451101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.8.218.254:37263)
0928 10:44:19.451379 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:19.451386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:19.451624 (+ 238us) server_negotiation.cc:407] Connection header received
0928 10:44:19.451776 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:19.451780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:19.451846 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:19.451931 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:19.453080 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33260: BlockingRecv error: recv got EOF from 127.0.0.1:33260 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 10:44:19.456481 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11588
W20260928 10:44:19.459062 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11588: Not authorized: Client connection negotiation failed: client connection to 127.8.218.193:38475: server requires authentication, but client does not have Kerberos credentials available
I20260928 10:44:19.459261 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 11588
I20260928 10:44:19.471992 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11722
W20260928 10:44:19.474879 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11722: Not authorized: Client connection negotiation failed: client connection to 127.8.218.194:36605: server requires authentication, but client does not have Kerberos credentials available
I20260928 10:44:19.475035 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 11722
I20260928 10:44:19.487797 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11855
W20260928 10:44:19.490793 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11855: Not authorized: Client connection negotiation failed: client connection to 127.8.218.195:38485: server requires authentication, but client does not have Kerberos credentials available
I20260928 10:44:19.490983 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 11855
I20260928 10:44:19.502918 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 11495
I20260928 10:44:19.504169 11557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:19.606189 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 11495
2026-09-28T10:44:19Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3525 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:19 dist-test-slave-mn38 krb5kdc[12000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:19 dist-test-slave-mn38 krb5kdc[12000](info): set up 2 sockets
Sep 28 10:44:19 dist-test-slave-mn38 krb5kdc[12000](info): commencing operation
krb5kdc: starting...
W20260928 10:44:20.626876 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:44:20 dist-test-slave-mn38 krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:44:20.835991 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:46437
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:46437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:21.190860 12013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:21.191191 12013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:21.191299 12013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:21.199470 12013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:21.199640 12013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:21.199692 12013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:21.199904 12013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:21.200013 12013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:21.211387 12013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:46437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:21.212877 12013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:21.214519 12013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:21.228073 12018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:21.227952 12019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:21.227988 12021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:21.229113 12013 server_base.cc:1034] running on GCE node
I20260928 10:44:21.230104 12013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:21.231773 12013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:21.233006 12013 hybrid_clock.cc:648] HybridClock initialized: now 1790592261232966 us; error 43 us; skew 500 ppm
I20260928 10:44:21.233423 12013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:21 dist-test-slave-mn38 krb5kdc[12000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592261, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:21.237658 12013 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:44:21.239346 12013 webserver.cc:460] Webserver started at http://127.8.218.254:45737/ using document root <none> and password file <none>
I20260928 10:44:21.240119 12013 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:21.240257 12013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:21.240577 12013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:21.243709 12013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "90bd33a3f8304dd0b83e473a2bbde972"
format_stamp: "Formatted at 2026-09-28 10:44:21 on dist-test-slave-mn38"
server_key: "eeb7e4b075f8cd2ee9a338651ff1182e"
server_key_iv: "ff48c2175c8519f42f6855964587a5e5"
server_key_version: "encryptionkey@0"
I20260928 10:44:21.244586 12013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "90bd33a3f8304dd0b83e473a2bbde972"
format_stamp: "Formatted at 2026-09-28 10:44:21 on dist-test-slave-mn38"
server_key: "eeb7e4b075f8cd2ee9a338651ff1182e"
server_key_iv: "ff48c2175c8519f42f6855964587a5e5"
server_key_version: "encryptionkey@0"
I20260928 10:44:21.250417 12013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 10:44:21.254644 12028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:21.256287 12013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:44:21.256485 12013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "90bd33a3f8304dd0b83e473a2bbde972"
format_stamp: "Formatted at 2026-09-28 10:44:21 on dist-test-slave-mn38"
server_key: "eeb7e4b075f8cd2ee9a338651ff1182e"
server_key_iv: "ff48c2175c8519f42f6855964587a5e5"
server_key_version: "encryptionkey@0"
I20260928 10:44:21.256701 12013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:21.286769 12013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:21.291780 12013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:21.292114 12013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:21.311038 12013 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:46437
I20260928 10:44:21.311055 12079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:46437 every 8 connection(s)
I20260928 10:44:21.312433 12013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:21.315912 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 12013
I20260928 10:44:21.316115 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:21.316361 12080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:21.316771 9067 external_mini_cluster.cc:1428] Setting key c49dce9a5fd2e704c389124f35db3204
I20260928 10:44:21.331388 12080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972: Bootstrap starting.
Sep 28 10:44:21 dist-test-slave-mn38 krb5kdc[12000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:21.335031 12080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:21.336218 12080 log.cc:824] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:21.339231 12080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972: No bootstrap required, opened a new log
I20260928 10:44:21.340847 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:21.318505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.8.218.254:46437)
0928 10:44:21.319292 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:21.319304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:21.319348 (+ 44us) server_negotiation.cc:407] Connection header received
0928 10:44:21.320973 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:21.320997 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:21.321475 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:21.321889 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:21.323886 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:21.325248 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:21.327038 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:21.327835 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:21.327982 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:21.333050 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:21.333063 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:21.333081 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:21.333128 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:21.336365 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:21.337227 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:21.337237 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:21.337244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:21.337460 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:21.337860 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:21.337864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:21.337866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:21.338443 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:21.338718 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:21.339208 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 10:44:21.339785 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":360,"threads_started":1}
I20260928 10:44:21.345192 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:44:21.348068 12080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd33a3f8304dd0b83e473a2bbde972" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46437 } }
I20260928 10:44:21.348541 12080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:21.348650 12080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bd33a3f8304dd0b83e473a2bbde972, State: Initialized, Role: FOLLOWER
I20260928 10:44:21.349387 12080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd33a3f8304dd0b83e473a2bbde972" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46437 } }
I20260928 10:44:21.349723 12080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:21.349862 12080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:21.350034 12080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:21.353076 12080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd33a3f8304dd0b83e473a2bbde972" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46437 } }
I20260928 10:44:21.353809 12080 leader_election.cc:302] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90bd33a3f8304dd0b83e473a2bbde972; no voters:
I20260928 10:44:21.354529 12080 leader_election.cc:288] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:21.354671 12086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:21.355918 12086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [term 1 LEADER]: Becoming Leader. State: Replica: 90bd33a3f8304dd0b83e473a2bbde972, State: Running, Role: LEADER
I20260928 10:44:21.356551 12086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd33a3f8304dd0b83e473a2bbde972" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46437 } }
I20260928 10:44:21.356998 12080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:21.359766 12088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90bd33a3f8304dd0b83e473a2bbde972. Latest consensus state: current_term: 1 leader_uuid: "90bd33a3f8304dd0b83e473a2bbde972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd33a3f8304dd0b83e473a2bbde972" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46437 } } }
I20260928 10:44:21.359634 12087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90bd33a3f8304dd0b83e473a2bbde972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd33a3f8304dd0b83e473a2bbde972" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46437 } } }
I20260928 10:44:21.360111 12087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:21.360111 12088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:21.366187 12092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:21.372839 12092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:21.400815 12092 catalog_manager.cc:1347] Generated new cluster ID: 00d844220cae4c42944dc395b7f1b292
I20260928 10:44:21.400955 12092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:21.411079 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:21.400335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.8.218.254:46437)
0928 10:44:21.400533 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:21.400540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:21.401269 (+ 729us) server_negotiation.cc:407] Connection header received
0928 10:44:21.401495 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:21.401505 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:21.401599 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:21.401722 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:21.403327 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:21.404453 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:21.406512 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:21.407210 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:21.407357 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:21.408897 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:21.408904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:21.408909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:21.410088 (+ 1179us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:21.410194 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:21.410228 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:21.410351 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:21.410554 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 10:44:21.410686 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:44:21.422294 12092 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:21.423975 12092 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:21.431972 12092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90bd33a3f8304dd0b83e473a2bbde972: Generated new TSK 0
I20260928 10:44:21.432945 12092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:44:21.441795 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12013
I20260928 10:44:21.443208 12075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:21.535924 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 12013
2026-09-28T10:44:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1924 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:21 dist-test-slave-mn38 krb5kdc[12114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:21 dist-test-slave-mn38 krb5kdc[12114](info): set up 2 sockets
Sep 28 10:44:21 dist-test-slave-mn38 krb5kdc[12114](info): commencing operation
krb5kdc: starting...
W20260928 10:44:22.546582 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:44:22 dist-test-slave-mn38 krb5kdc[12114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:44:22.747195 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:42927
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:40257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:42927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:23.088331 12127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:23.088667 12127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:23.088788 12127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:23.097661 12127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:23.097798 12127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:23.097877 12127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:23.097952 12127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:23.098012 12127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:23.108844 12127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:42927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:42927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:23.110348 12127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:23.111747 12127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:23.124197 12132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:23.124421 12135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:23.124184 12133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:23.125407 12127 server_base.cc:1034] running on GCE node
I20260928 10:44:23.126420 12127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:23.128106 12127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:23.129364 12127 hybrid_clock.cc:648] HybridClock initialized: now 1790592263129313 us; error 46 us; skew 500 ppm
I20260928 10:44:23.129798 12127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:23 dist-test-slave-mn38 krb5kdc[12114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592263, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:23.133342 12127 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:44:23.135233 12127 webserver.cc:460] Webserver started at http://127.8.218.254:43249/ using document root <none> and password file <none>
I20260928 10:44:23.135989 12127 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:23.136124 12127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:23.136444 12127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:23.139912 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "1ce7fb7b806043dd9f78b799706feb38"
format_stamp: "Formatted at 2026-09-28 10:44:23 on dist-test-slave-mn38"
server_key: "19c1ea979335aafab7bacf21a1fd6fd6"
server_key_iv: "7f9626da3317064856678d56aaae40f5"
server_key_version: "encryptionkey@0"
I20260928 10:44:23.140780 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "1ce7fb7b806043dd9f78b799706feb38"
format_stamp: "Formatted at 2026-09-28 10:44:23 on dist-test-slave-mn38"
server_key: "19c1ea979335aafab7bacf21a1fd6fd6"
server_key_iv: "7f9626da3317064856678d56aaae40f5"
server_key_version: "encryptionkey@0"
I20260928 10:44:23.146773 12127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 10:44:23.150583 12142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:23.151980 12127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:44:23.152171 12127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "1ce7fb7b806043dd9f78b799706feb38"
format_stamp: "Formatted at 2026-09-28 10:44:23 on dist-test-slave-mn38"
server_key: "19c1ea979335aafab7bacf21a1fd6fd6"
server_key_iv: "7f9626da3317064856678d56aaae40f5"
server_key_version: "encryptionkey@0"
I20260928 10:44:23.152379 12127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:23.173120 12127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:23.177759 12127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:23.178056 12127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:23.199241 12127 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:42927
I20260928 10:44:23.199277 12193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:42927 every 8 connection(s)
I20260928 10:44:23.200966 12127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:23.206429 12194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:23.206729 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 12127
I20260928 10:44:23.206960 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:23.207598 9067 external_mini_cluster.cc:1428] Setting key 33ebc0bdb91f80d09d90e50b8bd745fc
I20260928 10:44:23.222929 12194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38: Bootstrap starting.
Sep 28 10:44:23 dist-test-slave-mn38 krb5kdc[12114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:23.226642 12194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:23.228041 12194 log.cc:824] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:23.231702 12194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38: No bootstrap required, opened a new log
I20260928 10:44:23.236989 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:23.209903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.8.218.254:42927)
0928 10:44:23.211393 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:23.211411 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:23.211477 (+ 66us) server_negotiation.cc:407] Connection header received
0928 10:44:23.213947 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:23.213990 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:23.214706 (+ 716us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:23.215284 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:23.216810 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:23.219048 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:23.220620 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:23.221980 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:23.222230 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:23.227339 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:23.227352 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:23.227374 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:23.227434 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:23.231072 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:23.231974 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:23.231983 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:23.231990 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:23.232197 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:23.232645 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:23.232649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:23.232651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:23.233370 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:23.233705 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:23.234398 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 10:44:23.235218 (+ 820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1183,"thread_start_us":831,"threads_started":1}
I20260928 10:44:23.242057 12194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce7fb7b806043dd9f78b799706feb38" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42927 } }
I20260928 10:44:23.242477 12194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:23.242590 12194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce7fb7b806043dd9f78b799706feb38, State: Initialized, Role: FOLLOWER
I20260928 10:44:23.243211 12194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce7fb7b806043dd9f78b799706feb38" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42927 } }
I20260928 10:44:23.243431 12194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:23.243561 12194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:23.243708 12194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:23.246255 12194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce7fb7b806043dd9f78b799706feb38" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42927 } }
I20260928 10:44:23.246793 12194 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ce7fb7b806043dd9f78b799706feb38; no voters:
I20260928 10:44:23.247411 12194 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:23.247570 12199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:23.248179 12199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [term 1 LEADER]: Becoming Leader. State: Replica: 1ce7fb7b806043dd9f78b799706feb38, State: Running, Role: LEADER
I20260928 10:44:23.248769 12199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce7fb7b806043dd9f78b799706feb38" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42927 } }
I20260928 10:44:23.249508 12194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:23.251827 12200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ce7fb7b806043dd9f78b799706feb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce7fb7b806043dd9f78b799706feb38" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42927 } } }
I20260928 10:44:23.252139 12200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:23.251730 12201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce7fb7b806043dd9f78b799706feb38. Latest consensus state: current_term: 1 leader_uuid: "1ce7fb7b806043dd9f78b799706feb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce7fb7b806043dd9f78b799706feb38" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42927 } } }
I20260928 10:44:23.252426 12201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:23.253257 12204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:23.261121 12204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:23.273547 12204 catalog_manager.cc:1347] Generated new cluster ID: 0c33e4d0dff84005a51eb2b4978af478
I20260928 10:44:23.273687 12204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:23.291183 12204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:23.292860 12204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:23.309113 12204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce7fb7b806043dd9f78b799706feb38: Generated new TSK 0
I20260928 10:44:23.310091 12204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:44:23.362223 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42927
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:40257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:23.715822 12220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:23.716125 12220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:23.716217 12220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:23.723726 12220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:23.723861 12220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:23.723973 12220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:23.734130 12220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:23.735613 12220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:23.736969 12220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:23.750018 12226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:23.750097 12225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:23.750553 12228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:23.750849 12220 server_base.cc:1034] running on GCE node
I20260928 10:44:23.751497 12220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:23.752552 12220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:23.753768 12220 hybrid_clock.cc:648] HybridClock initialized: now 1790592263753713 us; error 50 us; skew 500 ppm
I20260928 10:44:23.754120 12220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:23 dist-test-slave-mn38 krb5kdc[12114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592263, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:23.757583 12220 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:44:23.759164 12220 webserver.cc:460] Webserver started at http://127.8.218.193:46305/ using document root <none> and password file <none>
I20260928 10:44:23.760073 12220 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:23.760213 12220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:23.760524 12220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:23.763753 12220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "6a419283c0ab4638b464c0ac688064f7"
format_stamp: "Formatted at 2026-09-28 10:44:23 on dist-test-slave-mn38"
server_key: "d61b1baff13bbdd4085b71e99177fbd7"
server_key_iv: "fab1b90b433df02ffb8b3401d6dea74c"
server_key_version: "encryptionkey@0"
I20260928 10:44:23.764647 12220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "6a419283c0ab4638b464c0ac688064f7"
format_stamp: "Formatted at 2026-09-28 10:44:23 on dist-test-slave-mn38"
server_key: "d61b1baff13bbdd4085b71e99177fbd7"
server_key_iv: "fab1b90b433df02ffb8b3401d6dea74c"
server_key_version: "encryptionkey@0"
I20260928 10:44:23.771108 12220 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:44:23.775496 12235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:23.777277 12220 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 10:44:23.777547 12220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "6a419283c0ab4638b464c0ac688064f7"
format_stamp: "Formatted at 2026-09-28 10:44:23 on dist-test-slave-mn38"
server_key: "d61b1baff13bbdd4085b71e99177fbd7"
server_key_iv: "fab1b90b433df02ffb8b3401d6dea74c"
server_key_version: "encryptionkey@0"
I20260928 10:44:23.777783 12220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:23.802204 12220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:23.810556 12220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:23.810899 12220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:23.811993 12220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:23.813880 12220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:23.813990 12220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:23.814090 12220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:23.814136 12220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:23.847432 12220 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:40989
I20260928 10:44:23.847496 12347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:40989 every 8 connection(s)
I20260928 10:44:23.849165 12220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:23.852319 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 12220
I20260928 10:44:23.852556 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:23.853317 9067 external_mini_cluster.cc:1428] Setting key fc313185db1197fe22715bc3bb5dd1fd
Sep 28 10:44:23 dist-test-slave-mn38 krb5kdc[12114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592263, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:44:23.873531 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:23.852058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:46399 (local address 127.8.218.254:42927)
0928 10:44:23.852560 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:23.852564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:23.854089 (+ 1525us) server_negotiation.cc:407] Connection header received
0928 10:44:23.856555 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:23.856561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:23.856670 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:23.856795 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:23.858534 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:23.859531 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:23.861476 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:23.861918 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:23.862047 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:23.866732 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:23.866739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:23.866743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:23.866794 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:23.869613 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:23.870755 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:23.870760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:23.870762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:23.870905 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:23.871433 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:23.871437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:23.871439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:23.871728 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:23.871853 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:23.872924 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0928 10:44:23.873151 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":281,"threads_started":1}
I20260928 10:44:23.874758 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:23.852905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42927 (local address 127.8.218.193:46399)
0928 10:44:23.853886 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:23.853936 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:23.856192 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:23.857057 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:23.857067 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:23.857694 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:23.858333 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:23.858353 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:23.859695 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:23.859699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:23.861214 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:23.861233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:23.862167 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:23.862174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:23.862478 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:23.863256 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:23.863285 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:23.866354 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:23.869809 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:23.869823 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:23.869846 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:23.870577 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:23.871020 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:23.871026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:23.871029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:23.871303 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:23.871868 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:23.871878 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:23.872398 (+ 520us) client_negotiation.cc:769] Sending connection context
0928 10:44:23.872745 (+ 347us) client_negotiation.cc:240] Negotiation successful
0928 10:44:23.873323 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":371,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
I20260928 10:44:23.876488 12348 heartbeater.cc:347] Connected to a master server at 127.8.218.254:42927
I20260928 10:44:23.876808 12348 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:23.877684 12348 heartbeater.cc:511] Master 127.8.218.254:42927 requested a full tablet report, sending...
I20260928 10:44:23.880151 12159 ts_manager.cc:194] Registered new tserver with Master: 6a419283c0ab4638b464c0ac688064f7 (127.8.218.193:40989)
I20260928 10:44:23.882174 12159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:46399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:44:23.905413 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42927
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:40257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:24.259845 12354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:24.260151 12354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:24.260249 12354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:24.267822 12354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:24.267989 12354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:24.268132 12354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:24.279134 12354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:24.280517 12354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:24.281903 12354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:24.294924 12360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:24.294891 12359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:24.294978 12362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:24.296041 12354 server_base.cc:1034] running on GCE node
I20260928 10:44:24.296698 12354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:24.297657 12354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:24.298866 12354 hybrid_clock.cc:648] HybridClock initialized: now 1790592264298804 us; error 50 us; skew 500 ppm
I20260928 10:44:24.299189 12354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:24 dist-test-slave-mn38 krb5kdc[12114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:24.302429 12354 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:44:24.303985 12354 webserver.cc:460] Webserver started at http://127.8.218.194:42291/ using document root <none> and password file <none>
I20260928 10:44:24.304698 12354 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:24.304805 12354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:24.305107 12354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:24.308102 12354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "b6085fcfe7954165a7bd03ecf1f644d8"
format_stamp: "Formatted at 2026-09-28 10:44:24 on dist-test-slave-mn38"
server_key: "8e9cd8d9e1c6c21086775ffb11e88460"
server_key_iv: "5f05da54cee34e3eccb3a4e5cd1dd2c5"
server_key_version: "encryptionkey@0"
I20260928 10:44:24.308869 12354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "b6085fcfe7954165a7bd03ecf1f644d8"
format_stamp: "Formatted at 2026-09-28 10:44:24 on dist-test-slave-mn38"
server_key: "8e9cd8d9e1c6c21086775ffb11e88460"
server_key_iv: "5f05da54cee34e3eccb3a4e5cd1dd2c5"
server_key_version: "encryptionkey@0"
I20260928 10:44:24.314260 12354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 10:44:24.318029 12369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:24.319705 12354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 10:44:24.319890 12354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "b6085fcfe7954165a7bd03ecf1f644d8"
format_stamp: "Formatted at 2026-09-28 10:44:24 on dist-test-slave-mn38"
server_key: "8e9cd8d9e1c6c21086775ffb11e88460"
server_key_iv: "5f05da54cee34e3eccb3a4e5cd1dd2c5"
server_key_version: "encryptionkey@0"
I20260928 10:44:24.320113 12354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:24.341050 12354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:24.345669 12354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:24.345949 12354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:24.346889 12354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:24.348390 12354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:24.348484 12354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:24.348592 12354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:24.348635 12354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:24.380259 12354 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:40047
I20260928 10:44:24.380292 12481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:40047 every 8 connection(s)
I20260928 10:44:24.381918 12354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:24.385694 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 12354
I20260928 10:44:24.385926 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:24.386510 9067 external_mini_cluster.cc:1428] Setting key a4b6f2f3cbece83aac5d75d13bc2ae4a
Sep 28 10:44:24 dist-test-slave-mn38 krb5kdc[12114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:44:24.408860 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:24.387272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:54367 (local address 127.8.218.254:42927)
0928 10:44:24.388910 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:24.388915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:24.388935 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:44:24.390512 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:24.390519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:24.390610 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:24.390737 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:24.392721 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.393766 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.396654 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.397143 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.397268 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:24.402084 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:24.402093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:24.402098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:24.402145 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:24.405055 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:24.406259 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:24.406264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:24.406266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:24.406407 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:24.406952 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:24.406956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:24.406958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:24.407238 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:24.407353 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:24.408261 (+ 908us) server_negotiation.cc:299] Negotiation successful
0928 10:44:24.408459 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1528,"thread_start_us":1432,"threads_started":1}
I20260928 10:44:24.410029 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:24.386914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42927 (local address 127.8.218.194:54367)
0928 10:44:24.388033 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:24.388091 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:24.390150 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:24.391020 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:24.391030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:24.391652 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:24.392444 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:24.392468 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.394028 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.394034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:24.396357 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:24.396384 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.397471 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.397482 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:24.397846 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:24.398682 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:24.398720 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:24.401682 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:24.405277 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:24.405289 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:24.405324 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:24.406111 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:24.406526 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:24.406532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:24.406537 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:24.406833 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:24.407380 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:24.407387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:24.407858 (+ 471us) client_negotiation.cc:769] Sending connection context
0928 10:44:24.408142 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 10:44:24.408597 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":407,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
I20260928 10:44:24.411808 12482 heartbeater.cc:347] Connected to a master server at 127.8.218.254:42927
I20260928 10:44:24.412141 12482 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:24.412988 12482 heartbeater.cc:511] Master 127.8.218.254:42927 requested a full tablet report, sending...
I20260928 10:44:24.414669 12159 ts_manager.cc:194] Registered new tserver with Master: b6085fcfe7954165a7bd03ecf1f644d8 (127.8.218.194:40047)
I20260928 10:44:24.415501 12159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:54367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:44:24.437383 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42927
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:40257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:24.785580 12488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:24.785887 12488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:24.785988 12488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:24.795749 12488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:24.795907 12488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:24.796041 12488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:24.807142 12488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:24.808508 12488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:24.809898 12488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:24.825096 12494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:24.825155 12496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:24.825125 12493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:24.826031 12488 server_base.cc:1034] running on GCE node
I20260928 10:44:24.826756 12488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:24.827736 12488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:24.828956 12488 hybrid_clock.cc:648] HybridClock initialized: now 1790592264828902 us; error 40 us; skew 500 ppm
I20260928 10:44:24.829298 12488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:24 dist-test-slave-mn38 krb5kdc[12114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:24.832804 12488 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:44:24.834519 12488 webserver.cc:460] Webserver started at http://127.8.218.195:37165/ using document root <none> and password file <none>
I20260928 10:44:24.835251 12488 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:24.835376 12488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:24.835711 12488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:24.839118 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "3a25a1f2fac7456bb8c80157c084765e"
format_stamp: "Formatted at 2026-09-28 10:44:24 on dist-test-slave-mn38"
server_key: "29f56a91e37567e4f60b6e1bd117a9c6"
server_key_iv: "c8529493be8d8442a7ad485eb49b213e"
server_key_version: "encryptionkey@0"
I20260928 10:44:24.839991 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "3a25a1f2fac7456bb8c80157c084765e"
format_stamp: "Formatted at 2026-09-28 10:44:24 on dist-test-slave-mn38"
server_key: "29f56a91e37567e4f60b6e1bd117a9c6"
server_key_iv: "c8529493be8d8442a7ad485eb49b213e"
server_key_version: "encryptionkey@0"
I20260928 10:44:24.846171 12488 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:44:24.850474 12503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:24.852433 12488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260928 10:44:24.852664 12488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "3a25a1f2fac7456bb8c80157c084765e"
format_stamp: "Formatted at 2026-09-28 10:44:24 on dist-test-slave-mn38"
server_key: "29f56a91e37567e4f60b6e1bd117a9c6"
server_key_iv: "c8529493be8d8442a7ad485eb49b213e"
server_key_version: "encryptionkey@0"
I20260928 10:44:24.852914 12488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:24.873109 12488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:24.877636 12488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:24.877941 12488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:24.878965 12488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:24.880460 12488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:24.880546 12488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:24.880677 12488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:24.880731 12488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:24.885715 12348 heartbeater.cc:503] Master 127.8.218.254:42927 was elected leader, sending a full tablet report...
I20260928 10:44:24.913403 12488 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:40731
I20260928 10:44:24.913467 12615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:40731 every 8 connection(s)
I20260928 10:44:24.914682 12488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:24.916757 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 12488
I20260928 10:44:24.916986 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:24.917622 9067 external_mini_cluster.cc:1428] Setting key 03df40bbc95f4dcedc214431fb3d83ec
Sep 28 10:44:24 dist-test-slave-mn38 krb5kdc[12114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:24.936569 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:24.917643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:42911 (local address 127.8.218.254:42927)
0928 10:44:24.918137 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:24.918143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:24.919029 (+ 886us) server_negotiation.cc:407] Connection header received
0928 10:44:24.921309 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:24.921314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:24.921401 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:24.921542 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:24.923206 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.924076 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.925976 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.926434 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.926571 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:24.930430 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:24.930436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:24.930439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:24.930478 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:24.933185 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:24.934152 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:24.934157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:24.934160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:24.934283 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:24.934717 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:24.934721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:24.934728 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:24.935009 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:24.935144 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:24.935949 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 10:44:24.936181 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":221,"threads_started":1}
I20260928 10:44:24.937654 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:24.918009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42927 (local address 127.8.218.195:42911)
0928 10:44:24.918851 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:24.918902 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:24.920982 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:24.921793 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:24.921803 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:24.922381 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:24.923016 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:24.923037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.924259 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.924264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:24.925773 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:24.925790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.926745 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.926751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:24.926983 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:24.927673 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:24.927702 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:24.930157 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:24.933339 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:24.933350 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:24.933371 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:24.934021 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:24.934392 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:24.934397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:24.934402 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:24.934599 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:24.935142 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:24.935151 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:24.935545 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 10:44:24.935797 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 10:44:24.936393 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":307,"threads_started":1}
I20260928 10:44:24.939129 12616 heartbeater.cc:347] Connected to a master server at 127.8.218.254:42927
I20260928 10:44:24.939443 12616 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:24.940217 12616 heartbeater.cc:511] Master 127.8.218.254:42927 requested a full tablet report, sending...
I20260928 10:44:24.941836 12159 ts_manager.cc:194] Registered new tserver with Master: 3a25a1f2fac7456bb8c80157c084765e (127.8.218.195:40731)
I20260928 10:44:24.942683 12159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:42911
I20260928 10:44:24.946902 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:44:24 dist-test-slave-mn38 krb5kdc[12114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 10:44:24 dist-test-slave-mn38 krb5kdc[12114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:24.999248 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:24.982472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.8.218.254:42927)
0928 10:44:24.982700 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:24.982706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:24.983149 (+ 443us) server_negotiation.cc:407] Connection header received
0928 10:44:24.983341 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:24.983345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:24.983429 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:24.983516 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:24.985058 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.985880 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.987480 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:24.988252 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:24.988403 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:24.993055 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:24.993063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:24.993066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:24.993114 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:24.996044 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:24.996903 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:24.996908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:24.996911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:24.997101 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:24.997786 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:24.997791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:24.997793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:24.998173 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:24.998383 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:24.998757 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 10:44:24.999005 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 10:44:24.999991 12159 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40028
I20260928 10:44:25.002466 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12220
Sep 28 10:44:25 dist-test-slave-mn38 krb5kdc[12114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:44:25.020658 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:25.003661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.8.218.193:40989)
0928 10:44:25.004346 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:25.004357 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:25.004387 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:44:25.004457 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:25.004464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:25.004693 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:25.004855 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:25.006217 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:25.007375 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:25.008752 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:25.009382 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:25.009616 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:25.014297 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:25.014318 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:25.014340 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:25.014391 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:25.017670 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:25.018625 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:25.018634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:25.018636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:25.018835 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:25.019207 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:25.019210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:25.019212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:25.019601 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:25.019870 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:25.020066 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 10:44:25.020277 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":359,"threads_started":1}
W20260928 10:44:25.021967 12343 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53322
W20260928 10:44:25.023124 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12220: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:25.023308 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 12220
I20260928 10:44:25.036316 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12354
Sep 28 10:44:25 dist-test-slave-mn38 krb5kdc[12114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:44:25.052078 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:25.037236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.8.218.194:40047)
0928 10:44:25.037837 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:25.037845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:25.037871 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:44:25.037938 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:25.037945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:25.038157 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:25.038292 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:25.039496 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:25.040514 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:25.041784 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:25.042391 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:25.042495 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:25.046373 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:25.046388 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:25.046404 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:25.046451 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:25.049249 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:25.050053 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:25.050061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:25.050063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:25.050198 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:25.050598 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:25.050602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:25.050604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:25.050988 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:25.051218 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:25.051489 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 10:44:25.051694 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":310,"threads_started":1}
W20260928 10:44:25.053376 12477 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49638
W20260928 10:44:25.054569 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:25.054723 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 12354
I20260928 10:44:25.067332 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12488
Sep 28 10:44:25 dist-test-slave-mn38 krb5kdc[12114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592264, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:44:25.085079 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:25.068347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.8.218.195:40731)
0928 10:44:25.068960 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:25.068967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:25.068998 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:44:25.069072 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:25.069079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:25.069297 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:25.069509 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:25.070950 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:25.072015 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:25.073341 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:25.074084 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:25.074215 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:25.078871 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:25.078889 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:25.078906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:25.078953 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:25.082069 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:25.082941 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:25.082947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:25.082950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:25.083127 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:25.083564 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:25.083567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:25.083569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:25.083939 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:25.084208 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:25.084527 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 10:44:25.084754 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":312,"threads_started":1}
W20260928 10:44:25.086558 12611 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41262
W20260928 10:44:25.087788 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12488: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:44:25.087947 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 12488
I20260928 10:44:25.101893 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12127
I20260928 10:44:25.103109 12189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:25.205940 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 12127
2026-09-28T10:44:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3675 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:25 dist-test-slave-mn38 krb5kdc[12634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:25 dist-test-slave-mn38 krb5kdc[12634](info): set up 2 sockets
Sep 28 10:44:25 dist-test-slave-mn38 krb5kdc[12634](info): commencing operation
krb5kdc: starting...
W20260928 10:44:26.227017 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:44:26 dist-test-slave-mn38 krb5kdc[12634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:44:26.435318 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38257
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:26.774811 12647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:26.775161 12647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:26.775296 12647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:26.782861 12647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:26.782989 12647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:26.783046 12647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:26.783124 12647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:26.783177 12647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:26.797529 12647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:26.798942 12647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:26.800271 12647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:26.812353 12652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:26.812407 12655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:26.812425 12653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:26.814121 12647 server_base.cc:1034] running on GCE node
I20260928 10:44:26.815253 12647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:26.817118 12647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:26.818413 12647 hybrid_clock.cc:648] HybridClock initialized: now 1790592266818348 us; error 55 us; skew 500 ppm
I20260928 10:44:26.818874 12647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:26 dist-test-slave-mn38 krb5kdc[12634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592266, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:26.822613 12647 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:44:26.824538 12647 webserver.cc:460] Webserver started at http://127.8.218.254:34057/ using document root <none> and password file <none>
I20260928 10:44:26.825340 12647 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:26.825517 12647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:26.825886 12647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:26.829329 12647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de"
format_stamp: "Formatted at 2026-09-28 10:44:26 on dist-test-slave-mn38"
server_key: "245841229a6251a778ec2833c007ab26"
server_key_iv: "55446d4cd2e5c51695d95954af0ec262"
server_key_version: "encryptionkey@0"
I20260928 10:44:26.830339 12647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de"
format_stamp: "Formatted at 2026-09-28 10:44:26 on dist-test-slave-mn38"
server_key: "245841229a6251a778ec2833c007ab26"
server_key_iv: "55446d4cd2e5c51695d95954af0ec262"
server_key_version: "encryptionkey@0"
I20260928 10:44:26.835934 12647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 10:44:26.839623 12662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:26.841063 12647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:44:26.841233 12647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de"
format_stamp: "Formatted at 2026-09-28 10:44:26 on dist-test-slave-mn38"
server_key: "245841229a6251a778ec2833c007ab26"
server_key_iv: "55446d4cd2e5c51695d95954af0ec262"
server_key_version: "encryptionkey@0"
I20260928 10:44:26.841490 12647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:26.856740 12647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:26.861183 12647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:26.861542 12647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:26.880036 12647 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:38257
I20260928 10:44:26.880069 12713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:38257 every 8 connection(s)
I20260928 10:44:26.881875 12647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:26.884948 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 12647
I20260928 10:44:26.885171 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:26.885986 9067 external_mini_cluster.cc:1428] Setting key 0e726b08b0487b8d52c60219ea2d810c
I20260928 10:44:26.887372 12714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:44:26 dist-test-slave-mn38 krb5kdc[12634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:26.901940 12714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de: Bootstrap starting.
I20260928 10:44:26.905412 12714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:26.906975 12714 log.cc:824] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:26.910792 12714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de: No bootstrap required, opened a new log
I20260928 10:44:26.910931 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:26.888146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.8.218.254:38257)
0928 10:44:26.888966 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:26.888978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:26.889020 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:44:26.890699 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:26.890726 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:26.891178 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:26.891557 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:26.893399 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:26.894929 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:26.896654 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:26.897360 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:26.897576 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:26.902252 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:26.902264 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:26.902288 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:26.902338 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:26.905481 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:26.906628 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:26.906636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:26.906645 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:26.906929 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:26.907442 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:26.907446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:26.907449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:26.908418 (+ 969us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:26.908702 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:26.909261 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 10:44:26.909809 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":363,"threads_started":1}
I20260928 10:44:26.920849 12714 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38257 } }
I20260928 10:44:26.921307 12714 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:26.921416 12714 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7fd7d2a6e447af9d22bc1a6a9764de, State: Initialized, Role: FOLLOWER
I20260928 10:44:26.922117 12714 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38257 } }
I20260928 10:44:26.922375 12714 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:26.922513 12714 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:26.922685 12714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:26.925339 12714 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38257 } }
I20260928 10:44:26.925966 12714 leader_election.cc:302] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad7fd7d2a6e447af9d22bc1a6a9764de; no voters:
I20260928 10:44:26.926589 12714 leader_election.cc:288] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:26.926770 12719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:26.927554 12719 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [term 1 LEADER]: Becoming Leader. State: Replica: ad7fd7d2a6e447af9d22bc1a6a9764de, State: Running, Role: LEADER
I20260928 10:44:26.928160 12719 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38257 } }
I20260928 10:44:26.929190 12714 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:26.931099 12721 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad7fd7d2a6e447af9d22bc1a6a9764de. Latest consensus state: current_term: 1 leader_uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38257 } } }
I20260928 10:44:26.931093 12720 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7fd7d2a6e447af9d22bc1a6a9764de" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38257 } } }
I20260928 10:44:26.931721 12721 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:26.931843 12720 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:26.933894 12724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:26.939074 12724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:26.952076 12724 catalog_manager.cc:1347] Generated new cluster ID: aac23f6937624659ab2b5e87f38f184b
I20260928 10:44:26.952327 12724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:26.967628 12724 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:26.969750 12724 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:26.989248 12724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad7fd7d2a6e447af9d22bc1a6a9764de: Generated new TSK 0
I20260928 10:44:26.990425 12724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:44:27.044899 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38257
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:27.419801 12740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:27.420125 12740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:27.420271 12740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:27.427934 12740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:27.428076 12740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:27.428228 12740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:27.438781 12740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:27.440193 12740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:27.441591 12740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:27.456516 12745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:27.456552 12748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:27.456588 12746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:27.457376 12740 server_base.cc:1034] running on GCE node
I20260928 10:44:27.458112 12740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:27.459170 12740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:27.460497 12740 hybrid_clock.cc:648] HybridClock initialized: now 1790592267460443 us; error 50 us; skew 500 ppm
I20260928 10:44:27.460901 12740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:27 dist-test-slave-mn38 krb5kdc[12634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592267, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:27.464383 12740 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:44:27.466116 12740 webserver.cc:460] Webserver started at http://127.8.218.193:34489/ using document root <none> and password file <none>
I20260928 10:44:27.466939 12740 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:27.467062 12740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:27.467363 12740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:27.470716 12740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "ef978dd901004decb1da2d839ceb711d"
format_stamp: "Formatted at 2026-09-28 10:44:27 on dist-test-slave-mn38"
server_key: "539b71877ba7346bb883a8d586dac74b"
server_key_iv: "1b9275197e50a52e033c86374a94a789"
server_key_version: "encryptionkey@0"
I20260928 10:44:27.471786 12740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "ef978dd901004decb1da2d839ceb711d"
format_stamp: "Formatted at 2026-09-28 10:44:27 on dist-test-slave-mn38"
server_key: "539b71877ba7346bb883a8d586dac74b"
server_key_iv: "1b9275197e50a52e033c86374a94a789"
server_key_version: "encryptionkey@0"
I20260928 10:44:27.478634 12740 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:44:27.482892 12755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:27.484596 12740 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:44:27.484848 12740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "ef978dd901004decb1da2d839ceb711d"
format_stamp: "Formatted at 2026-09-28 10:44:27 on dist-test-slave-mn38"
server_key: "539b71877ba7346bb883a8d586dac74b"
server_key_iv: "1b9275197e50a52e033c86374a94a789"
server_key_version: "encryptionkey@0"
I20260928 10:44:27.485081 12740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:27.500509 12740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:27.504989 12740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:27.505293 12740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:27.506340 12740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:27.507731 12740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:27.507812 12740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:27.507936 12740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:27.507990 12740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:27.541894 12740 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:42927
I20260928 10:44:27.541954 12867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:42927 every 8 connection(s)
I20260928 10:44:27.543298 12740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:27.548939 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 12740
I20260928 10:44:27.549160 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:27.549980 9067 external_mini_cluster.cc:1428] Setting key 79b15bad518d1e4192a982ffacf0ed61
Sep 28 10:44:27 dist-test-slave-mn38 krb5kdc[12634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592267, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:27.567728 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:27.546072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:47785 (local address 127.8.218.254:38257)
0928 10:44:27.546553 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:27.546558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:27.547667 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 10:44:27.550453 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:27.550458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:27.550549 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:27.550650 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:27.552349 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:27.553054 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:27.555558 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:27.555988 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:27.556127 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:27.560248 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:27.560254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:27.560258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:27.560296 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:27.563535 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:27.564577 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:27.564592 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:27.564596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:27.564766 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:27.565313 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:27.565320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:27.565324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:27.565727 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:27.565890 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:27.567045 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0928 10:44:27.567316 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":247,"threads_started":1}
I20260928 10:44:27.569002 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:27.546722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38257 (local address 127.8.218.193:47785)
0928 10:44:27.547494 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:27.547533 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:27.549957 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:27.550973 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:27.550984 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:27.551563 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:27.552172 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:27.552191 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:27.553235 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:27.553241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:27.555332 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:27.555356 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:27.556246 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:27.556253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:27.556579 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:27.557350 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:27.557380 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:27.559951 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:27.563752 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:27.563763 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:27.563784 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:27.564413 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:27.564890 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:27.564895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:27.564898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:27.565164 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:27.565989 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:27.565999 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:27.566515 (+ 516us) client_negotiation.cc:769] Sending connection context
0928 10:44:27.566865 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 10:44:27.567461 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":284,"threads_started":1}
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:44:27.571056 12868 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38257
I20260928 10:44:27.571501 12868 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:27.572584 12868 heartbeater.cc:511] Master 127.8.218.254:38257 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
I20260928 10:44:27.575407 12679 ts_manager.cc:194] Registered new tserver with Master: ef978dd901004decb1da2d839ceb711d (127.8.218.193:42927)
I20260928 10:44:27.577020 12679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:47785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:44:27.599047 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38257
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:27.945048 12874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:27.945372 12874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:27.945514 12874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:27.953089 12874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:27.953230 12874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:27.953387 12874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:27.963604 12874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:27.964952 12874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:27.966495 12874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:27.981277 12879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:27.981194 12880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:27.981194 12882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:27.982105 12874 server_base.cc:1034] running on GCE node
I20260928 10:44:27.982849 12874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:27.983950 12874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:27.987392 12874 hybrid_clock.cc:648] HybridClock initialized: now 1790592267987310 us; error 155 us; skew 500 ppm
I20260928 10:44:27.987819 12874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:27 dist-test-slave-mn38 krb5kdc[12634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592267, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:27.991724 12874 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:44:27.994189 12874 webserver.cc:460] Webserver started at http://127.8.218.194:44873/ using document root <none> and password file <none>
I20260928 10:44:27.995000 12874 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:27.995151 12874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:27.995476 12874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:27.998906 12874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "ae56efe28b9c4f5e9204f7e621180a88"
format_stamp: "Formatted at 2026-09-28 10:44:27 on dist-test-slave-mn38"
server_key: "fc9f73cff6f55f8488d83777a6fb9d86"
server_key_iv: "5271e44353ef4fd798508b3865a8a400"
server_key_version: "encryptionkey@0"
I20260928 10:44:27.999835 12874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "ae56efe28b9c4f5e9204f7e621180a88"
format_stamp: "Formatted at 2026-09-28 10:44:27 on dist-test-slave-mn38"
server_key: "fc9f73cff6f55f8488d83777a6fb9d86"
server_key_iv: "5271e44353ef4fd798508b3865a8a400"
server_key_version: "encryptionkey@0"
I20260928 10:44:28.006237 12874 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:44:28.011065 12889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:28.012969 12874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:44:28.013286 12874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "ae56efe28b9c4f5e9204f7e621180a88"
format_stamp: "Formatted at 2026-09-28 10:44:27 on dist-test-slave-mn38"
server_key: "fc9f73cff6f55f8488d83777a6fb9d86"
server_key_iv: "5271e44353ef4fd798508b3865a8a400"
server_key_version: "encryptionkey@0"
I20260928 10:44:28.013619 12874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:28.032241 12874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:28.037040 12874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:28.037354 12874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:28.038426 12874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:28.039894 12874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:28.039978 12874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:28.040099 12874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:28.040156 12874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:28.073268 12874 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:39271
I20260928 10:44:28.073297 13001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:39271 every 8 connection(s)
I20260928 10:44:28.074749 12874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:28.079665 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 12874
I20260928 10:44:28.079854 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:28.080354 9067 external_mini_cluster.cc:1428] Setting key d6b559e5dcdf75aea2f21d5d8cd1b7ac
Sep 28 10:44:28 dist-test-slave-mn38 krb5kdc[12634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592267, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:28.101751 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.077655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:49473 (local address 127.8.218.254:38257)
0928 10:44:28.078163 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:28.078170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:28.079882 (+ 1712us) server_negotiation.cc:407] Connection header received
0928 10:44:28.082365 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:28.082370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:28.082444 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:28.082534 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:28.084941 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.086067 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.087954 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.088640 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.088795 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.093518 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:28.093526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:28.093529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:28.093572 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:28.096296 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.097671 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.097677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.097681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.097884 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.099066 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.099071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.099074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.099676 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:28.099827 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:28.101149 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0928 10:44:28.101368 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:44:28.103154 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.078340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38257 (local address 127.8.218.194:49473)
0928 10:44:28.079617 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:28.079689 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:28.081773 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:28.082933 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:28.082944 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:28.083563 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:28.084671 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:28.084700 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.086289 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.086294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:28.087730 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:28.087749 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.089059 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.089068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:28.089375 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.090185 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:28.090218 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:28.093092 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:28.096552 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:28.096565 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:28.096592 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:28.097432 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:28.098040 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:28.098046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:28.098050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:28.098761 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:28.099883 (+ 1122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:28.099894 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:28.100585 (+ 691us) client_negotiation.cc:769] Sending connection context
0928 10:44:28.100977 (+ 392us) client_negotiation.cc:240] Negotiation successful
0928 10:44:28.101687 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":686,"threads_started":1}
I20260928 10:44:28.105865 13002 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38257
I20260928 10:44:28.106352 13002 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
I20260928 10:44:28.107442 13002 heartbeater.cc:511] Master 127.8.218.254:38257 requested a full tablet report, sending...
I20260928 10:44:28.110209 12679 ts_manager.cc:194] Registered new tserver with Master: ae56efe28b9c4f5e9204f7e621180a88 (127.8.218.194:39271)
I20260928 10:44:28.111478 12679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:49473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:44:28.144217 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38257
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:28.494899 13008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:28.495275 13008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:28.495374 13008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:28.504297 13008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:28.504446 13008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:28.504571 13008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:28.515413 13008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:28.516790 13008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:28.518260 13008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:28.532815 13016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:28.532771 13014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:28.532908 13013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:28.534085 13008 server_base.cc:1034] running on GCE node
I20260928 10:44:28.534736 13008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:28.535794 13008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:28.537024 13008 hybrid_clock.cc:648] HybridClock initialized: now 1790592268536955 us; error 61 us; skew 500 ppm
I20260928 10:44:28.537333 13008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:28 dist-test-slave-mn38 krb5kdc[12634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592268, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:28.540591 13008 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:44:28.550439 13008 webserver.cc:460] Webserver started at http://127.8.218.195:37273/ using document root <none> and password file <none>
I20260928 10:44:28.551221 13008 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:28.551378 13008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:28.551733 13008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:28.554909 13008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "53098792c5fb4d51b784695fe3cce166"
format_stamp: "Formatted at 2026-09-28 10:44:28 on dist-test-slave-mn38"
server_key: "2d7b968667c1180d0286f0bf7ff0ecb8"
server_key_iv: "1570b8f79c1be2fee2eb73e3a921d5dc"
server_key_version: "encryptionkey@0"
I20260928 10:44:28.555814 13008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "53098792c5fb4d51b784695fe3cce166"
format_stamp: "Formatted at 2026-09-28 10:44:28 on dist-test-slave-mn38"
server_key: "2d7b968667c1180d0286f0bf7ff0ecb8"
server_key_iv: "1570b8f79c1be2fee2eb73e3a921d5dc"
server_key_version: "encryptionkey@0"
I20260928 10:44:28.561429 13008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:44:28.565898 13023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:28.567488 13008 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:44:28.567731 13008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "53098792c5fb4d51b784695fe3cce166"
format_stamp: "Formatted at 2026-09-28 10:44:28 on dist-test-slave-mn38"
server_key: "2d7b968667c1180d0286f0bf7ff0ecb8"
server_key_iv: "1570b8f79c1be2fee2eb73e3a921d5dc"
server_key_version: "encryptionkey@0"
I20260928 10:44:28.567984 13008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:28.580669 12868 heartbeater.cc:503] Master 127.8.218.254:38257 was elected leader, sending a full tablet report...
I20260928 10:44:28.594029 13008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:28.599046 13008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:28.599331 13008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:28.600416 13008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:28.601951 13008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:28.602032 13008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:28.602149 13008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:28.602212 13008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:28.634231 13008 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:40809
I20260928 10:44:28.634313 13135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:40809 every 8 connection(s)
I20260928 10:44:28.635555 13008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:28.636052 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 13008
I20260928 10:44:28.636206 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:28.636749 9067 external_mini_cluster.cc:1428] Setting key 0751bcac4deb322728acda9555dac692
Sep 28 10:44:28 dist-test-slave-mn38 krb5kdc[12634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592268, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:28.656556 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.638102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:33541 (local address 127.8.218.254:38257)
0928 10:44:28.638553 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:28.638559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:28.639731 (+ 1172us) server_negotiation.cc:407] Connection header received
0928 10:44:28.641516 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:28.641520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:28.641593 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:28.641680 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:28.643090 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.643873 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.645607 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.646065 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.646168 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.650121 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:28.650129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:28.650134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:28.650182 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:28.653287 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.654234 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.654238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.654241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.654397 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.654920 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.654924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.654926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.655223 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:28.655354 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:28.656078 (+ 724us) server_negotiation.cc:299] Negotiation successful
0928 10:44:28.656269 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260928 10:44:28.657582 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.638814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38257 (local address 127.8.218.195:33541)
0928 10:44:28.639567 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:28.639627 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:28.641262 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:28.641871 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:28.641879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:28.642376 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:28.642927 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:28.642944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.644041 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.644052 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:28.645432 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:28.645477 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.646285 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.646291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:28.646492 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.647281 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:28.647307 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:28.649771 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:28.653552 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:28.653561 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:28.653580 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:28.654091 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:28.654557 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:28.654561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:28.654564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:28.654780 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:28.655344 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:28.655350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:28.655713 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 10:44:28.655959 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 10:44:28.656435 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":297,"threads_started":1}
I20260928 10:44:28.659087 13136 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38257
I20260928 10:44:28.659476 13136 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:28.660246 13136 heartbeater.cc:511] Master 127.8.218.254:38257 requested a full tablet report, sending...
I20260928 10:44:28.662010 12679 ts_manager.cc:194] Registered new tserver with Master: 53098792c5fb4d51b784695fe3cce166 (127.8.218.195:40809)
I20260928 10:44:28.662856 12679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:33541
I20260928 10:44:28.665589 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:44:28 dist-test-slave-mn38 krb5kdc[12634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:44:28 dist-test-slave-mn38 krb5kdc[12634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:44:28.715315 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.699062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.8.218.193:42927)
0928 10:44:28.699744 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:28.699754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:28.699786 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:44:28.699852 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:28.699859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:28.700080 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:28.700230 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:28.701887 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.703009 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.704292 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.704886 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.704989 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.708997 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:28.709012 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:28.709029 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:28.709068 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:28.712052 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.712838 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.712846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.712849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.712998 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.713419 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.713423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.713425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.713871 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:28.714296 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:28.714639 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 10:44:28.714937 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":342,"threads_started":1}
Sep 28 10:44:28 dist-test-slave-mn38 krb5kdc[12634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:28.733412 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.719893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.8.218.254:38257)
0928 10:44:28.720111 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:28.720117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:28.720207 (+ 90us) server_negotiation.cc:407] Connection header received
0928 10:44:28.720414 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:28.720420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:28.720494 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:28.720581 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:28.721877 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.722624 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.723843 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.724281 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.724393 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.728251 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:28.728257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:28.728260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:28.728298 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:28.730783 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.731470 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.731474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.731477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.731619 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.732034 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.732038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.732040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.732394 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:28.732604 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:28.732923 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 10:44:28.733103 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 10:44:28.734030 12679 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38076
I20260928 10:44:28.735184 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12740
I20260928 10:44:28.749224 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.736154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.8.218.193:42927)
0928 10:44:28.736355 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:28.736360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:28.736637 (+ 277us) server_negotiation.cc:407] Connection header received
0928 10:44:28.736773 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:28.736777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:28.736852 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:28.736938 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:28.738395 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.739276 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.740697 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.741134 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.741251 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.743911 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:28.743917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:28.743921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:28.743963 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:28.746719 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.747446 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.747450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.747452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.747581 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.747923 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.747926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.747929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.748184 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:28.748381 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:28.748726 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 10:44:28.748917 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 10:44:28.749935 12863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:28.836162 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 12740
I20260928 10:44:28.850425 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12874
Sep 28 10:44:28 dist-test-slave-mn38 krb5kdc[12634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:44:28.873520 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.851345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.8.218.194:39271)
0928 10:44:28.851894 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:28.851902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:28.851929 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:44:28.851992 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:28.851999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:28.852226 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:28.852366 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:28.853749 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.854788 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.856140 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.856743 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.856850 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.867537 (+ 10687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:28.867554 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:28.867572 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:28.867619 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:28.870508 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.871264 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.871272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.871274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.871410 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.871800 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.871805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.871807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.872207 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:28.872503 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:28.872860 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 10:44:28.873131 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":290,"threads_started":1}
I20260928 10:44:28.874804 12997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:28.957412 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 12874
I20260928 10:44:28.970798 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 13008
Sep 28 10:44:28 dist-test-slave-mn38 krb5kdc[12634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:44:28.989382 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:28.971989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.8.218.195:40809)
0928 10:44:28.972724 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:28.972731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:28.972763 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:44:28.972826 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:28.972832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:28.973073 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:28.973207 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:28.974642 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.975735 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.977191 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:28.977848 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:28.977954 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:28.982566 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:28.982582 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:28.982599 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:28.982649 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:28.986008 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.986981 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.986990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.986993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.987160 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:28.987637 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:28.987642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:28.987644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:28.988049 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:28.988307 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:28.988732 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 10:44:28.988965 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":413,"threads_started":1}
I20260928 10:44:28.990832 13131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:29.073953 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 13008
I20260928 10:44:29.087411 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 12647
I20260928 10:44:29.088810 12709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:29.194988 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 12647
2026-09-28T10:44:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3989 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:29 dist-test-slave-mn38 krb5kdc[13156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:29 dist-test-slave-mn38 krb5kdc[13156](info): set up 2 sockets
Sep 28 10:44:29 dist-test-slave-mn38 krb5kdc[13156](info): commencing operation
krb5kdc: starting...
W20260928 10:44:30.222808 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:44:30 dist-test-slave-mn38 krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:44:30.432966 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:33587
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:30.776132 13169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:30.776464 13169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:30.776583 13169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:30.783994 13169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:30.784121 13169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:30.784181 13169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:30.784260 13169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:30.784318 13169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:30.800729 13169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:33587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:30.802783 13169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:30.804570 13169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:30.818011 13174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:30.818054 13177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:30.818041 13175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:30.818868 13169 server_base.cc:1034] running on GCE node
I20260928 10:44:30.819734 13169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:30.821331 13169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:30.822597 13169 hybrid_clock.cc:648] HybridClock initialized: now 1790592270822535 us; error 56 us; skew 500 ppm
I20260928 10:44:30.823007 13169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:30 dist-test-slave-mn38 krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:30.827082 13169 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:44:30.829434 13169 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:30.829651 13169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:30.830154 13169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:30.835685 13169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "e9bc4b8165ca4165ba768e13028f520a"
format_stamp: "Formatted at 2026-09-28 10:44:30 on dist-test-slave-mn38"
server_key: "2a1a17c259385ec60b3adecba2d6ee4b"
server_key_iv: "55b847d48c47bb9a5cf7c8d3f9f8d964"
server_key_version: "encryptionkey@0"
I20260928 10:44:30.837306 13169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "e9bc4b8165ca4165ba768e13028f520a"
format_stamp: "Formatted at 2026-09-28 10:44:30 on dist-test-slave-mn38"
server_key: "2a1a17c259385ec60b3adecba2d6ee4b"
server_key_iv: "55b847d48c47bb9a5cf7c8d3f9f8d964"
server_key_version: "encryptionkey@0"
I20260928 10:44:30.849555 13169 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.006s
I20260928 10:44:30.856138 13183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:30.859122 13169 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260928 10:44:30.859468 13169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "e9bc4b8165ca4165ba768e13028f520a"
format_stamp: "Formatted at 2026-09-28 10:44:30 on dist-test-slave-mn38"
server_key: "2a1a17c259385ec60b3adecba2d6ee4b"
server_key_iv: "55b847d48c47bb9a5cf7c8d3f9f8d964"
server_key_version: "encryptionkey@0"
I20260928 10:44:30.859812 13169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:30.882709 13169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:30.888801 13169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:30.889110 13169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:30.907749 13234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:33587 every 8 connection(s)
I20260928 10:44:30.907876 13169 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:33587
I20260928 10:44:30.908859 13169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:30.913605 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 13169
I20260928 10:44:30.913825 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:30.914014 13235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:30.914484 9067 external_mini_cluster.cc:1428] Setting key 00303de8731274ec2110f4e188fcc461
I20260928 10:44:30.930652 13235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a: Bootstrap starting.
Sep 28 10:44:30 dist-test-slave-mn38 krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:30.934796 13235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:30.936020 13235 log.cc:824] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:30.939471 13235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a: No bootstrap required, opened a new log
I20260928 10:44:30.942097 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:30.916815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.8.218.254:33587)
0928 10:44:30.917775 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:30.917792 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:30.917842 (+ 50us) server_negotiation.cc:407] Connection header received
0928 10:44:30.919903 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:30.919936 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:30.920530 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:30.920975 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:30.923144 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:30.924797 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:30.926868 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:30.927838 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:30.928007 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:30.933065 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:30.933075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:30.933093 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:30.933154 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:30.936731 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:30.937628 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:30.937636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:30.937644 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:30.937839 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:30.938266 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:30.938270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:30.938272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:30.938898 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:30.939393 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:30.940016 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 10:44:30.940672 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":433,"threads_started":1}
I20260928 10:44:30.948331 13235 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bc4b8165ca4165ba768e13028f520a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33587 } }
I20260928 10:44:30.948791 13235 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:30.948887 13235 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bc4b8165ca4165ba768e13028f520a, State: Initialized, Role: FOLLOWER
I20260928 10:44:30.949765 13235 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bc4b8165ca4165ba768e13028f520a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33587 } }
I20260928 10:44:30.950100 13235 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:30.950309 13235 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:30.950490 13235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:30.953260 13235 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bc4b8165ca4165ba768e13028f520a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33587 } }
I20260928 10:44:30.953994 13235 leader_election.cc:302] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bc4b8165ca4165ba768e13028f520a; no voters:
I20260928 10:44:30.954825 13235 leader_election.cc:288] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:30.954898 13240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:30.955590 13240 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [term 1 LEADER]: Becoming Leader. State: Replica: e9bc4b8165ca4165ba768e13028f520a, State: Running, Role: LEADER
I20260928 10:44:30.956244 13240 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bc4b8165ca4165ba768e13028f520a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33587 } }
I20260928 10:44:30.957378 13235 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:30.959069 13242 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bc4b8165ca4165ba768e13028f520a. Latest consensus state: current_term: 1 leader_uuid: "e9bc4b8165ca4165ba768e13028f520a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bc4b8165ca4165ba768e13028f520a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33587 } } }
I20260928 10:44:30.959499 13242 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:30.960078 13241 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bc4b8165ca4165ba768e13028f520a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bc4b8165ca4165ba768e13028f520a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33587 } } }
I20260928 10:44:30.960537 13241 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:30.965817 13248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:30.972595 13248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:30.985162 13248 catalog_manager.cc:1347] Generated new cluster ID: d000e83d9ad44b638f6b97eaa059b81e
I20260928 10:44:30.985621 13248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:31.008122 13248 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:31.010032 13248 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:31.018097 13248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a: Generated new TSK 0
I20260928 10:44:31.019064 13248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:44:31.086445 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:33587
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:31.452167 13261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:31.452466 13261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:31.452562 13261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:31.460434 13261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:31.460575 13261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:31.460697 13261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:31.471527 13261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:31.472958 13261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:31.474380 13261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:31.489158 13266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:31.489199 13267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:31.489204 13269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:31.490134 13261 server_base.cc:1034] running on GCE node
I20260928 10:44:31.490893 13261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:31.492022 13261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:31.493248 13261 hybrid_clock.cc:648] HybridClock initialized: now 1790592271493183 us; error 54 us; skew 500 ppm
I20260928 10:44:31.493695 13261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:31 dist-test-slave-mn38 krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592271, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:31.497316 13261 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:44:31.498771 13261 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:31.498904 13261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:31.499233 13261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:31.502795 13261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76"
format_stamp: "Formatted at 2026-09-28 10:44:31 on dist-test-slave-mn38"
server_key: "b9608ffcfe3442e41c1114fcb06ceb18"
server_key_iv: "803ceddebada27a7e1944ae4de7aec2c"
server_key_version: "encryptionkey@0"
I20260928 10:44:31.503798 13261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76"
format_stamp: "Formatted at 2026-09-28 10:44:31 on dist-test-slave-mn38"
server_key: "b9608ffcfe3442e41c1114fcb06ceb18"
server_key_iv: "803ceddebada27a7e1944ae4de7aec2c"
server_key_version: "encryptionkey@0"
I20260928 10:44:31.510286 13261 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:44:31.514798 13275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:31.516619 13261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:44:31.516814 13261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76"
format_stamp: "Formatted at 2026-09-28 10:44:31 on dist-test-slave-mn38"
server_key: "b9608ffcfe3442e41c1114fcb06ceb18"
server_key_iv: "803ceddebada27a7e1944ae4de7aec2c"
server_key_version: "encryptionkey@0"
I20260928 10:44:31.517112 13261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:31.531086 13261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:31.535374 13261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:31.535625 13261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:31.536589 13261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:31.538090 13261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:31.538183 13261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:31.538269 13261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:31.538323 13261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:31.571004 13261 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:42407
I20260928 10:44:31.571097 13387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:42407 every 8 connection(s)
I20260928 10:44:31.572052 13261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:31.580380 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 13261
I20260928 10:44:31.580595 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:31.581399 9067 external_mini_cluster.cc:1428] Setting key 934aa5d6d41e68ce363b3ed69a46c132
Sep 28 10:44:31 dist-test-slave-mn38 krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592271, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:31.595620 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:31.574950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:53799 (local address 127.8.218.254:33587)
0928 10:44:31.575441 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:31.575446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:31.576530 (+ 1084us) server_negotiation.cc:407] Connection header received
0928 10:44:31.578615 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:31.578620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:31.578715 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:31.578815 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:31.580568 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:31.581722 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:31.584042 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:31.584663 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:31.584826 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:31.588779 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:31.588788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:31.588793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:31.588840 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:31.591872 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:31.593000 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:31.593005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:31.593008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:31.593153 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:31.593728 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:31.593732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:31.593734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:31.594046 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:31.594168 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:31.595125 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 10:44:31.595329 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
I20260928 10:44:31.597111 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:31.575589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:33587 (local address 127.8.218.193:53799)
0928 10:44:31.576333 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:31.576384 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:31.578275 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:31.579079 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:31.579091 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:31.579696 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:31.580340 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:31.580363 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:31.581934 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:31.581939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:31.583853 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:31.583871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:31.584983 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:31.584991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:31.585215 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:31.585949 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:31.585981 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:31.588446 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:31.592101 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:31.592113 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:31.592136 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:31.592801 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:31.593277 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:31.593282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:31.593285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:31.593614 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:31.594180 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:31.594190 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:31.594682 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 10:44:31.594985 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 10:44:31.595609 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":281,"threads_started":1}
I20260928 10:44:31.598845 13388 heartbeater.cc:347] Connected to a master server at 127.8.218.254:33587
I20260928 10:44:31.599212 13388 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:31.599936 13388 heartbeater.cc:511] Master 127.8.218.254:33587 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:44:31.602349 13200 ts_manager.cc:194] Registered new tserver with Master: 0a4c5cd0127a4afcacc2fd8e5ec0ae76 (127.8.218.193:42407)
I20260928 10:44:31.603919 13200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:53799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:44:31.628805 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:33587
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:31.967573 13394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:31.967893 13394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:31.968032 13394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:31.975895 13394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:31.976061 13394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:31.976231 13394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:31.986765 13394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:31.988436 13394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:31.990207 13394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:32.003774 13399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:32.003818 13400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:32.003839 13402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:32.004904 13394 server_base.cc:1034] running on GCE node
I20260928 10:44:32.005614 13394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:32.006618 13394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:32.007838 13394 hybrid_clock.cc:648] HybridClock initialized: now 1790592272007771 us; error 55 us; skew 500 ppm
I20260928 10:44:32.008159 13394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:32 dist-test-slave-mn38 krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592272, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:32.011531 13394 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:44:32.012915 13394 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:32.013029 13394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:32.013338 13394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:32.016525 13394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "dc1bfc93a35d437badd482d8f2c603a5"
format_stamp: "Formatted at 2026-09-28 10:44:32 on dist-test-slave-mn38"
server_key: "d41116abc7d5678342bbad27757ac652"
server_key_iv: "b617f7461acf312dca38b234c7ed91a6"
server_key_version: "encryptionkey@0"
I20260928 10:44:32.017534 13394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "dc1bfc93a35d437badd482d8f2c603a5"
format_stamp: "Formatted at 2026-09-28 10:44:32 on dist-test-slave-mn38"
server_key: "d41116abc7d5678342bbad27757ac652"
server_key_iv: "b617f7461acf312dca38b234c7ed91a6"
server_key_version: "encryptionkey@0"
I20260928 10:44:32.023334 13394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:44:32.027529 13408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:32.029675 13394 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:32.029922 13394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "dc1bfc93a35d437badd482d8f2c603a5"
format_stamp: "Formatted at 2026-09-28 10:44:32 on dist-test-slave-mn38"
server_key: "d41116abc7d5678342bbad27757ac652"
server_key_iv: "b617f7461acf312dca38b234c7ed91a6"
server_key_version: "encryptionkey@0"
I20260928 10:44:32.030306 13394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:32.046257 13394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:32.051129 13394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:32.051414 13394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:32.052456 13394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:32.054018 13394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:32.054122 13394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:32.054250 13394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:32.054320 13394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:32.084863 13394 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:41935
I20260928 10:44:32.084903 13520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:41935 every 8 connection(s)
I20260928 10:44:32.085868 13394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:32.088982 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 13394
I20260928 10:44:32.089150 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:32.089639 9067 external_mini_cluster.cc:1428] Setting key fe3b3c81edff4da96891870d5f50ec78
Sep 28 10:44:32 dist-test-slave-mn38 krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592272, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:44:32.110096 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.088903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:43219 (local address 127.8.218.254:33587)
0928 10:44:32.089109 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.089115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.090619 (+ 1504us) server_negotiation.cc:407] Connection header received
0928 10:44:32.092923 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.092928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.093005 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.093096 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:32.095407 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.096176 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.098570 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.099270 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.099426 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.103813 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:32.103819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:32.103822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:32.103867 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:32.106639 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:32.107605 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:32.107610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:32.107613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:32.107760 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:32.108226 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:32.108231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:32.108235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:32.108517 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:32.108651 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.109488 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.109747 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:44:32.111171 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.089529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:33587 (local address 127.8.218.194:43219)
0928 10:44:32.090435 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:32.090494 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:32.092551 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:32.093424 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:32.093460 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:32.094284 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:32.095139 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.095166 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.096371 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.096377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.098369 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.098387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.099548 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.099554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.099848 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.100581 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:32.100613 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:32.103429 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:32.106851 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:32.106864 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:32.106884 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:32.107437 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:32.107889 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:32.107893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:32.107895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:32.108097 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:32.108647 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:32.108658 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:32.109044 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 10:44:32.109312 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 10:44:32.109948 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":401,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
I20260928 10:44:32.112699 13521 heartbeater.cc:347] Connected to a master server at 127.8.218.254:33587
I20260928 10:44:32.113029 13521 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:32.113790 13521 heartbeater.cc:511] Master 127.8.218.254:33587 requested a full tablet report, sending...
I20260928 10:44:32.115325 13200 ts_manager.cc:194] Registered new tserver with Master: dc1bfc93a35d437badd482d8f2c603a5 (127.8.218.194:41935)
I20260928 10:44:32.116173 13200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:43219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:44:32.137156 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:33587
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:32.482312 13526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:32.482630 13526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:32.482728 13526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:32.490566 13526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:32.490710 13526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:32.490832 13526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:32.502362 13526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:32.503823 13526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:32.505697 13526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:32.520046 13532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:32.520125 13534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:32.520246 13531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:32.520948 13526 server_base.cc:1034] running on GCE node
I20260928 10:44:32.521667 13526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:32.522694 13526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:32.523943 13526 hybrid_clock.cc:648] HybridClock initialized: now 1790592272523875 us; error 57 us; skew 500 ppm
I20260928 10:44:32.524292 13526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:32 dist-test-slave-mn38 krb5kdc[13156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592272, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:32.527468 13526 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:44:32.528653 13526 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:32.528746 13526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:32.529012 13526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:32.532109 13526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "c36475f329ba4db2af485b0625d4a0ad"
format_stamp: "Formatted at 2026-09-28 10:44:32 on dist-test-slave-mn38"
server_key: "dc754187e16e37f16a7922cd0fc5f227"
server_key_iv: "da586b54d2668ff0f17ed2db0809578d"
server_key_version: "encryptionkey@0"
I20260928 10:44:32.532953 13526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "c36475f329ba4db2af485b0625d4a0ad"
format_stamp: "Formatted at 2026-09-28 10:44:32 on dist-test-slave-mn38"
server_key: "dc754187e16e37f16a7922cd0fc5f227"
server_key_iv: "da586b54d2668ff0f17ed2db0809578d"
server_key_version: "encryptionkey@0"
I20260928 10:44:32.538635 13526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:44:32.542761 13540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:32.544370 13526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:44:32.544595 13526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "c36475f329ba4db2af485b0625d4a0ad"
format_stamp: "Formatted at 2026-09-28 10:44:32 on dist-test-slave-mn38"
server_key: "dc754187e16e37f16a7922cd0fc5f227"
server_key_iv: "da586b54d2668ff0f17ed2db0809578d"
server_key_version: "encryptionkey@0"
I20260928 10:44:32.544979 13526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:32.568230 13526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:32.573031 13526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:32.573293 13526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:32.574376 13526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:32.575784 13526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:32.575875 13526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:32.576067 13526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:32.576138 13526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:32.607049 13526 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:39285
I20260928 10:44:32.607098 13652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:39285 every 8 connection(s)
I20260928 10:44:32.607642 13388 heartbeater.cc:503] Master 127.8.218.254:33587 was elected leader, sending a full tablet report...
I20260928 10:44:32.608031 13526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:32.617131 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 13526
I20260928 10:44:32.617307 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:32.617878 9067 external_mini_cluster.cc:1428] Setting key f65f6badcb441ddb405308e725efd80d
Sep 28 10:44:32 dist-test-slave-mn38 krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592272, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:32.631443 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.610732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:35677 (local address 127.8.218.254:33587)
0928 10:44:32.611139 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.611145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.612555 (+ 1410us) server_negotiation.cc:407] Connection header received
0928 10:44:32.614607 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.614612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.614698 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.614799 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:32.616690 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.617617 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.619923 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.620546 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.620677 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.624660 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:32.624666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:32.624669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:32.624710 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:32.627709 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:32.628731 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:32.628736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:32.628739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:32.628887 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:32.629426 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:32.629431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:32.629472 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:32.629788 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:32.629933 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.630778 (+ 845us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.631046 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":174,"threads_started":1}
I20260928 10:44:32.632606 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.611420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:33587 (local address 127.8.218.195:35677)
0928 10:44:32.612344 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:32.612407 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:32.614278 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:32.615067 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:32.615080 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:32.615814 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:32.616480 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.616502 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.617815 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.617821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.619707 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.619731 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.620818 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.620828 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.621067 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.621847 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:32.621874 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:32.624361 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:32.627943 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:32.627953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:32.627971 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:32.628543 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:32.629028 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:32.629032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:32.629035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:32.629284 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:32.629941 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:32.629951 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:32.630359 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 10:44:32.630646 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 10:44:32.631259 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":452,"threads_started":1}
I20260928 10:44:32.634169 13653 heartbeater.cc:347] Connected to a master server at 127.8.218.254:33587
I20260928 10:44:32.634531 13653 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:32.635262 13653 heartbeater.cc:511] Master 127.8.218.254:33587 requested a full tablet report, sending...
I20260928 10:44:32.637115 13200 ts_manager.cc:194] Registered new tserver with Master: c36475f329ba4db2af485b0625d4a0ad (127.8.218.195:39285)
I20260928 10:44:32.638044 13200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:35677
I20260928 10:44:32.646942 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:44:32.663785 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.651553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.8.218.254:33587)
0928 10:44:32.651783 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.651790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.652155 (+ 365us) server_negotiation.cc:407] Connection header received
0928 10:44:32.652300 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.652304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.652372 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.652453 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:32.653914 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.654618 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.655835 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.656256 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.656347 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.658683 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:32.658689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:32.658692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:32.658728 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:32.661282 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:32.661993 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:32.661997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:32.662000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:32.662149 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:32.662482 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:32.662485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:32.662487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:32.662813 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:32.662997 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.663278 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.663462 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 10:44:32.668146 13200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:32.675143 13200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:32.700843 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.689090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.8.218.193:42407)
0928 10:44:32.691200 (+ 2110us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.691211 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.691245 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:44:32.691355 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.691366 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.691682 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.692739 (+ 1057us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:32.693003 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.694813 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.698079 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.699662 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.699833 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.700056 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.700460 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.700598 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1918,"thread_start_us":1765,"threads_started":1}
I20260928 10:44:32.701016 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.688093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:42407 (local address 127.0.0.1:47958)
0928 10:44:32.690003 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:32.690024 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:32.690156 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:32.691986 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:32.691995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:32.692031 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:32.692479 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.692494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.695044 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.695052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.697791 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.697813 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.699926 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.699934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.700184 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.700244 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 10:44:32.700402 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 10:44:32.700632 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1771,"thread_start_us":11200,"threads_started":1}
I20260928 10:44:32.705622 13323 tablet_service.cc:1467] Processing CreateTablet for tablet c6d648852fdf412fa5a9ebbfb0ca1408 (DEFAULT_TABLE table=test-table [id=56a59dce161c4f73bb2d5d49e0de4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:32.707619 13323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d648852fdf412fa5a9ebbfb0ca1408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:32.709944 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.688616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:39285 (local address 127.0.0.1:56846)
0928 10:44:32.699565 (+ 10949us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:32.699586 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:32.699737 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:32.702372 (+ 2635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:32.702377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:32.702403 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:32.702745 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.702757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.704588 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.704593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.706727 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.706750 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.708844 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.708851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.709080 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.709125 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 10:44:32.709256 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 10:44:32.709419 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10811,"mutex_wait_us":1,"thread_start_us":1647,"threads_started":1}
I20260928 10:44:32.709964 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.689768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.8.218.195:39285)
0928 10:44:32.699276 (+ 9508us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.699286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.699707 (+ 421us) server_negotiation.cc:407] Connection header received
0928 10:44:32.701517 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.701526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.701852 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.702071 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:32.702943 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.704453 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.706991 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.708573 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.708749 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.709034 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.709335 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.709587 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9282,"thread_start_us":9099,"threads_started":1}
I20260928 10:44:32.710476 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.688034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:41935 (local address 127.0.0.1:48060)
0928 10:44:32.688832 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:32.688876 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:32.689089 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:32.703682 (+ 14593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:32.703687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:32.703719 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:32.704044 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.704056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.705805 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.705809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.707748 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.707764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.709643 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.709650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.709848 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.709888 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:44:32.709969 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 10:44:32.710124 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":734,"threads_started":1}
I20260928 10:44:32.711664 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.688583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.8.218.194:41935)
0928 10:44:32.702096 (+ 13513us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.702104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.702133 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:44:32.702226 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.702234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.703441 (+ 1207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.703666 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:32.704215 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.705637 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.707979 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.709344 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.710907 (+ 1563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.711104 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.711180 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.711329 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13347,"thread_start_us":14537,"threads_started":1}
I20260928 10:44:32.714811 13588 tablet_service.cc:1467] Processing CreateTablet for tablet c6d648852fdf412fa5a9ebbfb0ca1408 (DEFAULT_TABLE table=test-table [id=56a59dce161c4f73bb2d5d49e0de4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:32.715679 13456 tablet_service.cc:1467] Processing CreateTablet for tablet c6d648852fdf412fa5a9ebbfb0ca1408 (DEFAULT_TABLE table=test-table [id=56a59dce161c4f73bb2d5d49e0de4ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:32.716228 13588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d648852fdf412fa5a9ebbfb0ca1408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:32.717031 13456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d648852fdf412fa5a9ebbfb0ca1408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:32.723465 13672 tablet_bootstrap.cc:492] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Bootstrap starting.
I20260928 10:44:32.727474 13672 tablet_bootstrap.cc:654] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:32.728878 13672 log.cc:824] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:32.729362 13673 tablet_bootstrap.cc:492] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: Bootstrap starting.
I20260928 10:44:32.733516 13673 tablet_bootstrap.cc:654] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:32.734412 13675 tablet_bootstrap.cc:492] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: Bootstrap starting.
I20260928 10:44:32.735214 13673 log.cc:824] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:32.738093 13675 tablet_bootstrap.cc:654] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:32.738497 13672 tablet_bootstrap.cc:492] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: No bootstrap required, opened a new log
I20260928 10:44:32.738611 13673 tablet_bootstrap.cc:492] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: No bootstrap required, opened a new log
I20260928 10:44:32.738809 13672 ts_tablet_manager.cc:1397] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260928 10:44:32.738834 13673 ts_tablet_manager.cc:1397] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 10:44:32.739171 13675 log.cc:824] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:32.741977 13675 tablet_bootstrap.cc:492] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: No bootstrap required, opened a new log
I20260928 10:44:32.742298 13675 ts_tablet_manager.cc:1397] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260928 10:44:32.749552 13672 raft_consensus.cc:348] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.749552 13673 raft_consensus.cc:348] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.750079 13672 raft_consensus.cc:374] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:32.750079 13673 raft_consensus.cc:374] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:32.749835 13675 raft_consensus.cc:348] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.750186 13673 raft_consensus.cc:729] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36475f329ba4db2af485b0625d4a0ad, State: Initialized, Role: FOLLOWER
I20260928 10:44:32.750218 13672 raft_consensus.cc:729] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4c5cd0127a4afcacc2fd8e5ec0ae76, State: Initialized, Role: FOLLOWER
I20260928 10:44:32.750226 13675 raft_consensus.cc:374] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:32.750435 13675 raft_consensus.cc:729] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1bfc93a35d437badd482d8f2c603a5, State: Initialized, Role: FOLLOWER
I20260928 10:44:32.750924 13673 consensus_queue.cc:260] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.750926 13672 consensus_queue.cc:260] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.751029 13675 consensus_queue.cc:260] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.752187 13521 heartbeater.cc:503] Master 127.8.218.254:33587 was elected leader, sending a full tablet report...
I20260928 10:44:32.752985 13653 heartbeater.cc:503] Master 127.8.218.254:33587 was elected leader, sending a full tablet report...
I20260928 10:44:32.753238 13672 ts_tablet_manager.cc:1428] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260928 10:44:32.753750 13675 ts_tablet_manager.cc:1428] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260928 10:44:32.755046 13678 raft_consensus.cc:482] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:32.755661 13673 ts_tablet_manager.cc:1428] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260928 10:44:32.755398 13678 raft_consensus.cc:504] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.758387 13678 leader_election.cc:288] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc1bfc93a35d437badd482d8f2c603a5 (127.8.218.194:41935), 0a4c5cd0127a4afcacc2fd8e5ec0ae76 (127.8.218.193:42407)
I20260928 10:44:32.764923 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.759242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:33199 (local address 127.8.218.193:42407)
0928 10:44:32.759715 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.759720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.759733 (+ 13us) server_negotiation.cc:407] Connection header received
0928 10:44:32.759781 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.759785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.759835 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.759904 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:32.760445 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.761361 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.763234 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.764070 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.764230 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.764303 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.764564 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.764684 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331}
I20260928 10:44:32.764948 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.758441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:52489 (local address 127.8.218.194:41935)
0928 10:44:32.758612 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.758617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.758634 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:44:32.758694 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.758700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.758770 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.758858 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:32.760075 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.761066 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.762935 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.763850 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.763975 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.764030 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.764531 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.764713 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 10:44:32.764973 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.757983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:41935 (local address 127.8.218.195:52489)
0928 10:44:32.758184 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:32.758204 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:32.758310 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:32.759241 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:32.759246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:32.759277 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:32.759464 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.759476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.761203 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.761209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.762762 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.762777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.764023 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.764105 (+ 82us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.764360 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.764396 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:44:32.764473 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 10:44:32.764622 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":23}
I20260928 10:44:32.765506 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.758807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:42407 (local address 127.8.218.195:33199)
0928 10:44:32.759454 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:32.759471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:32.759568 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:32.759936 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:32.759941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:32.759965 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:32.760287 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.760299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.761528 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.761532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.763044 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:32.763060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.764265 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.764269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:32.764411 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.764434 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 10:44:32.764528 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 10:44:32.764669 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":25,"thread_start_us":2465,"threads_started":1}
I20260928 10:44:32.766151 13476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d648852fdf412fa5a9ebbfb0ca1408" candidate_uuid: "c36475f329ba4db2af485b0625d4a0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc1bfc93a35d437badd482d8f2c603a5" is_pre_election: true
I20260928 10:44:32.766425 13343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d648852fdf412fa5a9ebbfb0ca1408" candidate_uuid: "c36475f329ba4db2af485b0625d4a0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" is_pre_election: true
I20260928 10:44:32.766800 13476 raft_consensus.cc:2454] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c36475f329ba4db2af485b0625d4a0ad in term 0.
I20260928 10:44:32.766928 13343 raft_consensus.cc:2454] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c36475f329ba4db2af485b0625d4a0ad in term 0.
I20260928 10:44:32.767577 13542 leader_election.cc:302] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4c5cd0127a4afcacc2fd8e5ec0ae76, c36475f329ba4db2af485b0625d4a0ad; no voters:
I20260928 10:44:32.768049 13678 raft_consensus.cc:2781] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:32.768290 13678 raft_consensus.cc:482] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:32.768384 13678 raft_consensus.cc:3037] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:32.770774 13678 raft_consensus.cc:504] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.771592 13678 leader_election.cc:288] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [CANDIDATE]: Term 1 election: Requested vote from peers dc1bfc93a35d437badd482d8f2c603a5 (127.8.218.194:41935), 0a4c5cd0127a4afcacc2fd8e5ec0ae76 (127.8.218.193:42407)
I20260928 10:44:32.772190 13476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d648852fdf412fa5a9ebbfb0ca1408" candidate_uuid: "c36475f329ba4db2af485b0625d4a0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc1bfc93a35d437badd482d8f2c603a5"
I20260928 10:44:32.772221 13343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d648852fdf412fa5a9ebbfb0ca1408" candidate_uuid: "c36475f329ba4db2af485b0625d4a0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76"
I20260928 10:44:32.772459 13476 raft_consensus.cc:3037] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:32.772459 13343 raft_consensus.cc:3037] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:32.775931 13343 raft_consensus.cc:2454] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c36475f329ba4db2af485b0625d4a0ad in term 1.
I20260928 10:44:32.775931 13476 raft_consensus.cc:2454] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c36475f329ba4db2af485b0625d4a0ad in term 1.
I20260928 10:44:32.776594 13541 leader_election.cc:302] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c36475f329ba4db2af485b0625d4a0ad, dc1bfc93a35d437badd482d8f2c603a5; no voters:
I20260928 10:44:32.776984 13678 raft_consensus.cc:2781] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:32.777571 13678 raft_consensus.cc:686] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 1 LEADER]: Becoming Leader. State: Replica: c36475f329ba4db2af485b0625d4a0ad, State: Running, Role: LEADER
I20260928 10:44:32.778168 13678 consensus_queue.cc:237] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } }
I20260928 10:44:32.783274 13199 catalog_manager.cc:5495] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to c36475f329ba4db2af485b0625d4a0ad (127.8.218.195). New cstate: current_term: 1 leader_uuid: "c36475f329ba4db2af485b0625d4a0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c36475f329ba4db2af485b0625d4a0ad" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 39285 } health_report { overall_health: HEALTHY } } }
I20260928 10:44:32.806190 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.800727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.8.218.195:39285)
0928 10:44:32.800924 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.800931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.801014 (+ 83us) server_negotiation.cc:407] Connection header received
0928 10:44:32.801253 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.801259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.801359 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.801533 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:32.801885 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.802606 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.803953 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.804342 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.804453 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.804962 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:32.805567 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:32.805715 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.805789 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.805877 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:44:32.815428 13343 raft_consensus.cc:1261] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 1 FOLLOWER]: Refusing update from remote peer c36475f329ba4db2af485b0625d4a0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:32.815428 13476 raft_consensus.cc:1261] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 1 FOLLOWER]: Refusing update from remote peer c36475f329ba4db2af485b0625d4a0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:32.816632 13682 consensus_queue.cc:1035] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4c5cd0127a4afcacc2fd8e5ec0ae76" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:32.817019 13678 consensus_queue.cc:1035] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc1bfc93a35d437badd482d8f2c603a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:44:32.824241 13389 tablet.cc:2369] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:44:32.835866 13687 mvcc.cc:205] Tried to move back new op lower bound from 7334265949438930944 to 7334265949303320576. Current Snapshot: MvccSnapshot[applied={T|T < 7334265949438930944}]
I20260928 10:44:32.846860 13686 mvcc.cc:205] Tried to move back new op lower bound from 7334265949438930944 to 7334265949303320576. Current Snapshot: MvccSnapshot[applied={T|T < 7334265949438930944}]
I20260928 10:44:32.859086 13199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:32.859391 13199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:32.863583 13199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9bc4b8165ca4165ba768e13028f520a: Sending DeleteTablet for 3 replicas of tablet c6d648852fdf412fa5a9ebbfb0ca1408
I20260928 10:44:32.864866 13456 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d648852fdf412fa5a9ebbfb0ca1408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:32 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:48060
I20260928 10:44:32.864858 13323 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d648852fdf412fa5a9ebbfb0ca1408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:32 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:47958
I20260928 10:44:32.865690 13588 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d648852fdf412fa5a9ebbfb0ca1408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:32 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:56846
I20260928 10:44:32.866566 13696 tablet_replica.cc:330] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: stopping tablet replica
I20260928 10:44:32.866988 13696 raft_consensus.cc:2229] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:32.867439 13696 raft_consensus.cc:2258] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:32.867974 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 13261
I20260928 10:44:32.868249 13697 tablet_replica.cc:330] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: stopping tablet replica
I20260928 10:44:32.868697 13697 raft_consensus.cc:2229] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:32.869475 13697 raft_consensus.cc:2258] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:32.870222 13696 ts_tablet_manager.cc:1905] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:32.870261 13695 tablet_replica.cc:330] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: stopping tablet replica
I20260928 10:44:32.870666 13695 raft_consensus.cc:2229] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:32.871198 13695 raft_consensus.cc:2258] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:32.873418 13697 ts_tablet_manager.cc:1905] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:32.877674 13696 ts_tablet_manager.cc:1918] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:32.878001 13696 log.cc:1196] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/c6d648852fdf412fa5a9ebbfb0ca1408
I20260928 10:44:32.878670 13696 ts_tablet_manager.cc:1939] T c6d648852fdf412fa5a9ebbfb0ca1408 P 0a4c5cd0127a4afcacc2fd8e5ec0ae76: Deleting consensus metadata
I20260928 10:44:32.880802 13185 catalog_manager.cc:4841] TS 0a4c5cd0127a4afcacc2fd8e5ec0ae76 (127.8.218.193:42407): tablet c6d648852fdf412fa5a9ebbfb0ca1408 (table test-table [id=56a59dce161c4f73bb2d5d49e0de4ae3]) successfully deleted
I20260928 10:44:32.881238 13697 ts_tablet_manager.cc:1918] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:32.881430 13697 log.cc:1196] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/c6d648852fdf412fa5a9ebbfb0ca1408
I20260928 10:44:32.883131 13695 ts_tablet_manager.cc:1905] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:32.883917 13697 ts_tablet_manager.cc:1939] T c6d648852fdf412fa5a9ebbfb0ca1408 P c36475f329ba4db2af485b0625d4a0ad: Deleting consensus metadata
I20260928 10:44:32.886014 13187 catalog_manager.cc:4841] TS c36475f329ba4db2af485b0625d4a0ad (127.8.218.195:39285): tablet c6d648852fdf412fa5a9ebbfb0ca1408 (table test-table [id=56a59dce161c4f73bb2d5d49e0de4ae3]) successfully deleted
I20260928 10:44:32.889899 13695 ts_tablet_manager.cc:1918] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:32.890090 13695 log.cc:1196] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/c6d648852fdf412fa5a9ebbfb0ca1408
Sep 28 10:44:32 dist-test-slave-mn38 krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:44:32.890666 13695 ts_tablet_manager.cc:1939] T c6d648852fdf412fa5a9ebbfb0ca1408 P dc1bfc93a35d437badd482d8f2c603a5: Deleting consensus metadata
I20260928 10:44:32.892547 13184 catalog_manager.cc:4841] TS dc1bfc93a35d437badd482d8f2c603a5 (127.8.218.194:41935): tablet c6d648852fdf412fa5a9ebbfb0ca1408 (table test-table [id=56a59dce161c4f73bb2d5d49e0de4ae3]) successfully deleted
I20260928 10:44:32.899282 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:32.870181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.8.218.193:42407)
0928 10:44:32.870361 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:32.870368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:32.880033 (+ 9665us) server_negotiation.cc:407] Connection header received
0928 10:44:32.880537 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:32.880543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:32.880636 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:32.880709 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:32.882475 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.883554 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.885527 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:32.886205 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:32.886328 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:32.891684 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:32.891707 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:32.891729 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:32.891793 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:32.895784 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:32.896674 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:32.896685 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:32.896688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:32.896880 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:32.897370 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:32.897377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:32.897380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:32.898005 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:32.898360 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:32.898655 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 10:44:32.898897 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:44:32.900039 13383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:33.023204 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 13261
I20260928 10:44:33.037886 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 13394
Sep 28 10:44:33 dist-test-slave-mn38 krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:44:33.055691 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:33.038885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.8.218.194:41935)
0928 10:44:33.039075 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:33.039081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:33.039104 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:44:33.039230 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:33.039236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:33.039340 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:33.039460 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:33.040776 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:33.041976 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:33.043376 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:33.044000 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:33.044134 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:33.048620 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:33.048642 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:33.048664 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:33.048717 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:33.052409 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:33.053172 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:33.053183 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:33.053187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:33.053354 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:33.053951 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:33.053957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:33.053961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:33.054472 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:33.054759 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:33.055086 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 10:44:33.055331 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:44:33.056546 13516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:33.147003 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 13394
I20260928 10:44:33.161777 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 13526
Sep 28 10:44:33 dist-test-slave-mn38 krb5kdc[13156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:44:33.180763 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:33.162639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.8.218.195:39285)
0928 10:44:33.162836 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:33.162840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:33.162858 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:44:33.163011 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:33.163015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:33.163091 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:33.163172 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:33.164855 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:33.165878 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:33.168035 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:33.168782 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:33.168917 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:33.174628 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:33.174643 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:33.174660 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:33.174712 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:33.177791 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:33.178555 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:33.178562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:33.178564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:33.178740 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:33.179176 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:33.179180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:33.179182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:33.179591 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:33.179897 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:33.180192 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 10:44:33.180421 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 10:44:33.181537 13648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:33.307070 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 13526
I20260928 10:44:33.323696 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 13169
I20260928 10:44:33.324862 13230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:33.433938 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 13169
2026-09-28T10:44:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4237 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T10:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:33Z Disabled control of system clock
I20260928 10:44:33.496788 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:44505
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:44505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 10:44:33.837853 13706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:33.838179 13706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:33.838269 13706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:33.845978 13706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:33.846118 13706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:33.846156 13706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:33.846208 13706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:33.846252 13706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:33.856407 13706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:44505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:44505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:33.857829 13706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:33.860206 13706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:33.873333 13714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:33.873353 13711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:33.873330 13712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:33.874114 13706 server_base.cc:1034] running on GCE node
I20260928 10:44:33.874977 13706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:33.876537 13706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:33.877812 13706 hybrid_clock.cc:648] HybridClock initialized: now 1790592273877762 us; error 40 us; skew 500 ppm
I20260928 10:44:33.878226 13706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:44:33.886324 13706 webserver.cc:460] Webserver started at http://127.8.218.254:45877/ using document root <none> and password file <none>
I20260928 10:44:33.887135 13706 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:33.887265 13706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:33.887590 13706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:33.890879 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "3e89da4638d44d3d9eba4debfc220bee"
format_stamp: "Formatted at 2026-09-28 10:44:33 on dist-test-slave-mn38"
server_key: "709bb0d3a48bfc6d67c9d1005a467492"
server_key_iv: "5e929180e5186329484652e3216b1326"
server_key_version: "encryptionkey@0"
I20260928 10:44:33.891790 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "3e89da4638d44d3d9eba4debfc220bee"
format_stamp: "Formatted at 2026-09-28 10:44:33 on dist-test-slave-mn38"
server_key: "709bb0d3a48bfc6d67c9d1005a467492"
server_key_iv: "5e929180e5186329484652e3216b1326"
server_key_version: "encryptionkey@0"
I20260928 10:44:33.897794 13706 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:44:33.901798 13720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:33.903362 13706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:44:33.903578 13706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "3e89da4638d44d3d9eba4debfc220bee"
format_stamp: "Formatted at 2026-09-28 10:44:33 on dist-test-slave-mn38"
server_key: "709bb0d3a48bfc6d67c9d1005a467492"
server_key_iv: "5e929180e5186329484652e3216b1326"
server_key_version: "encryptionkey@0"
I20260928 10:44:33.903829 13706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:33.910797 13706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:33.911726 13706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:33.912034 13706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:33.930327 13706 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:44505
I20260928 10:44:33.930356 13771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:44505 every 8 connection(s)
I20260928 10:44:33.931833 13706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:33.935796 13772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:33.941785 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 13706
I20260928 10:44:33.942005 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:33.942760 9067 external_mini_cluster.cc:1428] Setting key 5ab19af98ea1d6474de3fb2a706c5eb8
I20260928 10:44:33.947008 13772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee: Bootstrap starting.
I20260928 10:44:33.950685 13772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:33.951473 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:33.944898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.8.218.254:44505)
0928 10:44:33.945953 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:33.945966 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:33.946007 (+ 41us) server_negotiation.cc:407] Connection header received
0928 10:44:33.947678 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:33.947704 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:33.948154 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:33.948525 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:33.949658 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:33.949715 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:33.949730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:33.949746 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:33.949812 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:33.949830 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:33.949974 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:33.950132 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 10:44:33.950401 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":350,"threads_started":1}
I20260928 10:44:33.951867 13772 log.cc:824] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:33.955081 13772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee: No bootstrap required, opened a new log
I20260928 10:44:33.962534 13772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e89da4638d44d3d9eba4debfc220bee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44505 } }
I20260928 10:44:33.963029 13772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:33.963110 13772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e89da4638d44d3d9eba4debfc220bee, State: Initialized, Role: FOLLOWER
I20260928 10:44:33.963680 13772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e89da4638d44d3d9eba4debfc220bee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44505 } }
I20260928 10:44:33.963896 13772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:33.964058 13772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:33.964251 13772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:33.966564 13772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e89da4638d44d3d9eba4debfc220bee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44505 } }
I20260928 10:44:33.967137 13772 leader_election.cc:302] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e89da4638d44d3d9eba4debfc220bee; no voters:
I20260928 10:44:33.967744 13772 leader_election.cc:288] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:33.967880 13777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:33.968827 13777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [term 1 LEADER]: Becoming Leader. State: Replica: 3e89da4638d44d3d9eba4debfc220bee, State: Running, Role: LEADER
I20260928 10:44:33.969538 13777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e89da4638d44d3d9eba4debfc220bee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44505 } }
I20260928 10:44:33.969906 13772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:33.972402 13779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e89da4638d44d3d9eba4debfc220bee. Latest consensus state: current_term: 1 leader_uuid: "3e89da4638d44d3d9eba4debfc220bee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e89da4638d44d3d9eba4debfc220bee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44505 } } }
I20260928 10:44:33.972848 13779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:33.972779 13778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e89da4638d44d3d9eba4debfc220bee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e89da4638d44d3d9eba4debfc220bee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44505 } } }
I20260928 10:44:33.973199 13778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:33.976104 13783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:33.981823 13783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:33.993096 13783 catalog_manager.cc:1347] Generated new cluster ID: f2f39335dcb046029fb5c15aff758a61
I20260928 10:44:33.993223 13783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:34.004405 13783 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:34.004873 13783 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:34.015728 13783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee: Generated new TSK 0
I20260928 10:44:34.016675 13783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:44:34.029210 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44505
--builtin_ntp_servers=127.8.218.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 10:44:34.387935 13796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:34.388238 13796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:34.388350 13796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:34.395951 13796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:34.396075 13796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:34.396219 13796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:34.405864 13796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:34.407178 13796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:34.409071 13796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:34.422482 13801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:34.422515 13802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:34.422449 13804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:34.423357 13796 server_base.cc:1034] running on GCE node
I20260928 10:44:34.423979 13796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:34.424993 13796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:34.426232 13796 hybrid_clock.cc:648] HybridClock initialized: now 1790592274426158 us; error 68 us; skew 500 ppm
I20260928 10:44:34.426580 13796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:44:34.428941 13796 webserver.cc:460] Webserver started at http://127.8.218.193:37431/ using document root <none> and password file <none>
I20260928 10:44:34.429733 13796 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:34.429845 13796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:34.430151 13796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:34.433269 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "5ccd16872ce04ac4a59937a43ae740b4"
format_stamp: "Formatted at 2026-09-28 10:44:34 on dist-test-slave-mn38"
server_key: "afea7c932556435cb7da2ac4a57df3e4"
server_key_iv: "9922fbeb070f5e5af354176ab5e3a89b"
server_key_version: "encryptionkey@0"
I20260928 10:44:34.434203 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "5ccd16872ce04ac4a59937a43ae740b4"
format_stamp: "Formatted at 2026-09-28 10:44:34 on dist-test-slave-mn38"
server_key: "afea7c932556435cb7da2ac4a57df3e4"
server_key_iv: "9922fbeb070f5e5af354176ab5e3a89b"
server_key_version: "encryptionkey@0"
I20260928 10:44:34.439772 13796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 10:44:34.443773 13810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:34.445535 13796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:44:34.445724 13796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "5ccd16872ce04ac4a59937a43ae740b4"
format_stamp: "Formatted at 2026-09-28 10:44:34 on dist-test-slave-mn38"
server_key: "afea7c932556435cb7da2ac4a57df3e4"
server_key_iv: "9922fbeb070f5e5af354176ab5e3a89b"
server_key_version: "encryptionkey@0"
I20260928 10:44:34.445974 13796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:34.452571 13796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:34.453634 13796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:34.453922 13796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:34.455008 13796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:34.456600 13796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:34.456701 13796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:34.456815 13796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:34.456882 13796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:34.489156 13796 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:34443
I20260928 10:44:34.489223 13922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:34443 every 8 connection(s)
I20260928 10:44:34.490571 13796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:34.492025 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 13796
I20260928 10:44:34.492208 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:34.492792 9067 external_mini_cluster.cc:1428] Setting key 85c056b90f7c69769df000ee8f57d9ce
I20260928 10:44:34.496169 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44505
--builtin_ntp_servers=127.8.218.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 10:44:34.500097 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:34.493164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:45493 (local address 127.8.218.254:44505)
0928 10:44:34.493739 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:34.493745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:34.495087 (+ 1342us) server_negotiation.cc:407] Connection header received
0928 10:44:34.497551 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:34.497557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:34.497617 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:34.497719 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:34.499306 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:34.499311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:34.499315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:34.499335 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:34.499416 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:34.499430 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:34.499547 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:34.499779 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 10:44:34.499845 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":337,"threads_started":1}
I20260928 10:44:34.501631 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:34.493837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44505 (local address 127.8.218.193:45493)
0928 10:44:34.494874 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:34.494986 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:34.497153 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:34.498020 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:34.498034 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:34.498183 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:34.498903 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:44:34.498906 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:34.498969 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:44:34.499088 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:34.499531 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:34.499541 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:34.499548 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:44:34.499832 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 10:44:34.500147 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":285,"threads_started":1}
I20260928 10:44:34.503612 13923 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44505
I20260928 10:44:34.503968 13923 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:34.504618 13923 heartbeater.cc:511] Master 127.8.218.254:44505 requested a full tablet report, sending...
I20260928 10:44:34.507305 13737 ts_manager.cc:194] Registered new tserver with Master: 5ccd16872ce04ac4a59937a43ae740b4 (127.8.218.193:34443)
W20260928 10:44:34.848697 13927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:34.849018 13927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:34.849109 13927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:34.856726 13927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:34.856841 13927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:34.856983 13927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:34.866334 13927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:34.867647 13927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:34.869673 13927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:34.884047 13932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:34.884104 13935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:34.884104 13933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:34.884634 13927 server_base.cc:1034] running on GCE node
I20260928 10:44:34.885397 13927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:34.886607 13927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:34.887861 13927 hybrid_clock.cc:648] HybridClock initialized: now 1790592274887810 us; error 40 us; skew 500 ppm
I20260928 10:44:34.888232 13927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:44:34.890847 13927 webserver.cc:460] Webserver started at http://127.8.218.194:39207/ using document root <none> and password file <none>
I20260928 10:44:34.891741 13927 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:34.891849 13927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:34.892179 13927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:34.895505 13927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "05f4c5ca75654865be60bbb84d926aa0"
format_stamp: "Formatted at 2026-09-28 10:44:34 on dist-test-slave-mn38"
server_key: "01e3420159949f9a2a6292240db394d8"
server_key_iv: "b25225b73238cd5c3721f21d84497b26"
server_key_version: "encryptionkey@0"
I20260928 10:44:34.896653 13927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "05f4c5ca75654865be60bbb84d926aa0"
format_stamp: "Formatted at 2026-09-28 10:44:34 on dist-test-slave-mn38"
server_key: "01e3420159949f9a2a6292240db394d8"
server_key_iv: "b25225b73238cd5c3721f21d84497b26"
server_key_version: "encryptionkey@0"
I20260928 10:44:34.902987 13927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260928 10:44:34.907706 13941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:34.909466 13927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 10:44:34.909670 13927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "05f4c5ca75654865be60bbb84d926aa0"
format_stamp: "Formatted at 2026-09-28 10:44:34 on dist-test-slave-mn38"
server_key: "01e3420159949f9a2a6292240db394d8"
server_key_iv: "b25225b73238cd5c3721f21d84497b26"
server_key_version: "encryptionkey@0"
I20260928 10:44:34.909917 13927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:34.916585 13927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:34.917588 13927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:34.917886 13927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:34.918910 13927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:34.920599 13927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:34.920713 13927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:34.920812 13927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:34.920879 13927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:34.955766 13927 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:42481
I20260928 10:44:34.955818 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:42481 every 8 connection(s)
I20260928 10:44:34.957384 13927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:34.963009 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 13927
I20260928 10:44:34.963214 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:34.963725 9067 external_mini_cluster.cc:1428] Setting key 2bc9682b73beb5b00048b80e2799bef2
I20260928 10:44:34.966747 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:34.960242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:34209 (local address 127.8.218.254:44505)
0928 10:44:34.960456 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:34.960463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:34.962082 (+ 1619us) server_negotiation.cc:407] Connection header received
0928 10:44:34.964219 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:34.964225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:34.964310 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:34.964402 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:34.966012 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:34.966018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:34.966022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:34.966043 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:34.966082 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:34.966097 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:34.966237 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:34.966401 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 10:44:34.966470 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:44:34.966890 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44505
--builtin_ntp_servers=127.8.218.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 10:44:34.967885 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:34.960836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44505 (local address 127.8.218.194:34209)
0928 10:44:34.961912 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:34.961956 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:34.963883 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:34.964688 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:34.964698 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:34.964803 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:34.965603 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:44:34.965606 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:34.965662 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:44:34.965824 (+ 162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:34.966201 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:34.966209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:34.966215 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:44:34.966451 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 10:44:34.966710 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":307,"threads_started":1}
I20260928 10:44:34.969528 14054 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44505
I20260928 10:44:34.969928 14054 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:34.970685 14054 heartbeater.cc:511] Master 127.8.218.254:44505 requested a full tablet report, sending...
I20260928 10:44:34.972679 13737 ts_manager.cc:194] Registered new tserver with Master: 05f4c5ca75654865be60bbb84d926aa0 (127.8.218.194:42481)
W20260928 10:44:35.313406 14057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:35.313810 14057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:35.313951 14057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:35.321539 14057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:35.321666 14057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:35.321801 14057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:35.331066 14057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:35.332381 14057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:35.334210 14057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:35.347978 14062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:35.347958 14063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:35.347958 14065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:35.348377 14057 server_base.cc:1034] running on GCE node
I20260928 10:44:35.349326 14057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:35.350559 14057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:35.351796 14057 hybrid_clock.cc:648] HybridClock initialized: now 1790592275351724 us; error 64 us; skew 500 ppm
I20260928 10:44:35.352128 14057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:44:35.354581 14057 webserver.cc:460] Webserver started at http://127.8.218.195:41515/ using document root <none> and password file <none>
I20260928 10:44:35.355276 14057 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:35.355366 14057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:35.355682 14057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:35.358688 14057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "34f071a3ad8e473488f3219543bc7d8a"
format_stamp: "Formatted at 2026-09-28 10:44:35 on dist-test-slave-mn38"
server_key: "ad1e5b65948cb10ad683d83d4de273f6"
server_key_iv: "f033d1d48d5de4e1591d0cb7145a6e74"
server_key_version: "encryptionkey@0"
I20260928 10:44:35.359524 14057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "34f071a3ad8e473488f3219543bc7d8a"
format_stamp: "Formatted at 2026-09-28 10:44:35 on dist-test-slave-mn38"
server_key: "ad1e5b65948cb10ad683d83d4de273f6"
server_key_iv: "f033d1d48d5de4e1591d0cb7145a6e74"
server_key_version: "encryptionkey@0"
I20260928 10:44:35.364675 14057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 10:44:35.368431 14071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:35.369973 14057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:44:35.370148 14057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "34f071a3ad8e473488f3219543bc7d8a"
format_stamp: "Formatted at 2026-09-28 10:44:35 on dist-test-slave-mn38"
server_key: "ad1e5b65948cb10ad683d83d4de273f6"
server_key_iv: "f033d1d48d5de4e1591d0cb7145a6e74"
server_key_version: "encryptionkey@0"
I20260928 10:44:35.370379 14057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:35.376506 14057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:35.377509 14057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:35.377801 14057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:35.378849 14057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:35.380419 14057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:35.380523 14057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:35.380640 14057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:35.380707 14057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:35.412204 14057 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:33817
I20260928 10:44:35.412248 14183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:33817 every 8 connection(s)
I20260928 10:44:35.413693 14057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:35.419729 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 14057
I20260928 10:44:35.419896 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:35.420341 9067 external_mini_cluster.cc:1428] Setting key 8734714fbea69b20fca9f21767c859dc
I20260928 10:44:35.422438 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.416387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:41775 (local address 127.8.218.254:44505)
0928 10:44:35.416637 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.416643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.418283 (+ 1640us) server_negotiation.cc:407] Connection header received
0928 10:44:35.420023 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.420029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.420108 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.420193 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.421725 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.421731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.421736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.421754 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.421786 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.421798 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.421902 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.422096 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.422162 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 10:44:35.423496 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.416994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44505 (local address 127.8.218.195:41775)
0928 10:44:35.418126 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.418170 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:35.419707 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:35.420483 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:35.420496 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:35.420611 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:35.421284 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:44:35.421287 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:35.421331 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:44:35.421545 (+ 214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:35.421881 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:35.421891 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:35.421899 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:44:35.422182 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 10:44:35.422495 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":360,"threads_started":1}
I20260928 10:44:35.424809 14184 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44505
I20260928 10:44:35.425133 14184 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:35.425863 14184 heartbeater.cc:511] Master 127.8.218.254:44505 requested a full tablet report, sending...
I20260928 10:44:35.427480 13737 ts_manager.cc:194] Registered new tserver with Master: 34f071a3ad8e473488f3219543bc7d8a (127.8.218.195:33817)
I20260928 10:44:35.436254 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:44:35.443102 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.440702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.8.218.254:44505)
0928 10:44:35.440926 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.440934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.441278 (+ 344us) server_negotiation.cc:407] Connection header received
0928 10:44:35.441551 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.441555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.441613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.441696 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.442618 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.442621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.442625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.442637 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.442663 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.442673 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.442763 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.442794 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.442837 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:44:35.447057 13737 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:35.454165 13737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:35.483624 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.467282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.8.218.194:42481)
0928 10:44:35.478751 (+ 11469us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.478761 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.479031 (+ 270us) server_negotiation.cc:407] Connection header received
0928 10:44:35.479338 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.479347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.479660 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.479809 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.482526 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.482537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.482556 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.482581 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.482655 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.482679 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.482806 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.483154 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.483280 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11028,"spinlock_wait_cycles":511872,"thread_start_us":10901,"threads_started":1}
I20260928 10:44:35.484864 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.467037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:42481 (local address 127.0.0.1:46060)
0928 10:44:35.478855 (+ 11818us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.478897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:35.479167 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:35.481022 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:35.481032 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:35.481159 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:35.482112 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:44:35.482117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:35.482154 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:44:35.482266 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:35.482912 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:35.482926 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:35.482938 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:44:35.484200 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0928 10:44:35.484337 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11506,"mutex_wait_us":27,"thread_start_us":11898,"threads_started":1}
I20260928 10:44:35.484828 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.467231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33817 (local address 127.0.0.1:39996)
0928 10:44:35.478987 (+ 11756us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.479008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:35.479239 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:35.482619 (+ 3380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:35.482625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:35.482658 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:35.483433 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:44:35.483437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:35.483458 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:44:35.483606 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:35.484113 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:35.484117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:35.484122 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 10:44:35.484258 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 10:44:35.484320 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11614,"thread_start_us":11150,"threads_started":1}
I20260928 10:44:35.485265 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.468020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.8.218.195:33817)
0928 10:44:35.478175 (+ 10155us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.478186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.479192 (+ 1006us) server_negotiation.cc:407] Connection header received
0928 10:44:35.480913 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.480925 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.481205 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.481369 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.483843 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.483853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.483880 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.483903 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.483968 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.483994 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.484178 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.484246 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.484353 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9920,"thread_start_us":9721,"threads_started":1}
I20260928 10:44:35.489187 14119 tablet_service.cc:1467] Processing CreateTablet for tablet 122c8c61c68b429aa1d101d7594320ca (DEFAULT_TABLE table=test-table [id=89a22040e49640829127b4c673eab9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:35.489183 13989 tablet_service.cc:1467] Processing CreateTablet for tablet 122c8c61c68b429aa1d101d7594320ca (DEFAULT_TABLE table=test-table [id=89a22040e49640829127b4c673eab9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:35.490330 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.479401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.0.0.1:35246)
0928 10:44:35.484061 (+ 4660us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.484081 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:35.484200 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:35.485976 (+ 1776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:35.485981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:35.486011 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:35.488905 (+ 2894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:44:35.488909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:35.488930 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:44:35.489003 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:35.489626 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:35.489631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:35.489637 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:44:35.489786 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 10:44:35.489855 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4505,"thread_start_us":3809,"threads_started":1}
I20260928 10:44:35.490904 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122c8c61c68b429aa1d101d7594320ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:35.490904 13989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122c8c61c68b429aa1d101d7594320ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:35.491281 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.479699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.8.218.193:34443)
0928 10:44:35.483840 (+ 4141us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.483849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.484327 (+ 478us) server_negotiation.cc:407] Connection header received
0928 10:44:35.484400 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.484408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.484679 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.484826 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.489231 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.489241 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.489256 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.489283 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.489355 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.489379 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.489565 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.490754 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.490867 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3969,"thread_start_us":3862,"threads_started":1}
I20260928 10:44:35.495548 13858 tablet_service.cc:1467] Processing CreateTablet for tablet 122c8c61c68b429aa1d101d7594320ca (DEFAULT_TABLE table=test-table [id=89a22040e49640829127b4c673eab9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:35.497220 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122c8c61c68b429aa1d101d7594320ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:35.506135 14202 tablet_bootstrap.cc:492] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: Bootstrap starting.
I20260928 10:44:35.508260 14203 tablet_bootstrap.cc:492] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: Bootstrap starting.
I20260928 10:44:35.510919 13923 heartbeater.cc:503] Master 127.8.218.254:44505 was elected leader, sending a full tablet report...
I20260928 10:44:35.512164 14203 tablet_bootstrap.cc:654] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:35.512670 14204 tablet_bootstrap.cc:492] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4: Bootstrap starting.
I20260928 10:44:35.513231 14202 tablet_bootstrap.cc:654] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:35.514155 14203 log.cc:824] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:35.514672 14202 log.cc:824] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:35.516069 14204 tablet_bootstrap.cc:654] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:35.517302 14204 log.cc:824] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:35.517413 14203 tablet_bootstrap.cc:492] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: No bootstrap required, opened a new log
I20260928 10:44:35.517740 14203 ts_tablet_manager.cc:1397] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260928 10:44:35.517957 14202 tablet_bootstrap.cc:492] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: No bootstrap required, opened a new log
I20260928 10:44:35.518253 14202 ts_tablet_manager.cc:1397] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260928 10:44:35.521225 14204 tablet_bootstrap.cc:492] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4: No bootstrap required, opened a new log
I20260928 10:44:35.521584 14204 ts_tablet_manager.cc:1397] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 10:44:35.525767 14203 raft_consensus.cc:348] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.526234 14203 raft_consensus.cc:374] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:35.526360 14203 raft_consensus.cc:729] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4c5ca75654865be60bbb84d926aa0, State: Initialized, Role: FOLLOWER
I20260928 10:44:35.527043 14203 consensus_queue.cc:260] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.528179 14054 heartbeater.cc:503] Master 127.8.218.254:44505 was elected leader, sending a full tablet report...
I20260928 10:44:35.528576 14202 raft_consensus.cc:348] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.529001 14203 ts_tablet_manager.cc:1428] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260928 10:44:35.529039 14202 raft_consensus.cc:374] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:35.529134 14202 raft_consensus.cc:729] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f071a3ad8e473488f3219543bc7d8a, State: Initialized, Role: FOLLOWER
I20260928 10:44:35.529932 14202 consensus_queue.cc:260] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.531113 14184 heartbeater.cc:503] Master 127.8.218.254:44505 was elected leader, sending a full tablet report...
I20260928 10:44:35.531805 14202 ts_tablet_manager.cc:1428] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260928 10:44:35.533391 14204 raft_consensus.cc:348] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.534049 14204 raft_consensus.cc:374] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:35.534168 14204 raft_consensus.cc:729] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccd16872ce04ac4a59937a43ae740b4, State: Initialized, Role: FOLLOWER
I20260928 10:44:35.534818 14204 consensus_queue.cc:260] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.536784 14204 ts_tablet_manager.cc:1428] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 10:44:35.594496 14208 raft_consensus.cc:482] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:35.594858 14208 raft_consensus.cc:504] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.596546 14208 leader_election.cc:288] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34f071a3ad8e473488f3219543bc7d8a (127.8.218.195:33817), 5ccd16872ce04ac4a59937a43ae740b4 (127.8.218.193:34443)
I20260928 10:44:35.600195 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.597025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:45905 (local address 127.8.218.195:33817)
0928 10:44:35.597239 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.597246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.598129 (+ 883us) server_negotiation.cc:407] Connection header received
0928 10:44:35.598414 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.598418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.598475 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.598575 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.599583 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.599589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.599594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.599610 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.599643 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.599655 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.599765 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.599848 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.599915 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:44:35.600344 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.596973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33817 (local address 127.8.218.194:45905)
0928 10:44:35.598004 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.598030 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:35.598247 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:35.598614 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:35.598619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:35.598642 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:35.599352 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:44:35.599356 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:35.599374 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:44:35.599424 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:35.599742 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:35.599747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:35.599752 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 10:44:35.599887 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 10:44:35.599936 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":356,"threads_started":1}
I20260928 10:44:35.600481 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.598048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:51915 (local address 127.8.218.193:34443)
0928 10:44:35.598259 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.598264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.598471 (+ 207us) server_negotiation.cc:407] Connection header received
0928 10:44:35.598586 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.598591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.598643 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.598724 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.599829 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.599834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.599837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.599853 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.599884 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.599926 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.600031 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.600156 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.600217 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 10:44:35.600677 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.597761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.8.218.194:51915)
0928 10:44:35.598355 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.598370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:35.598482 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:35.598776 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:35.598780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:35.598805 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:35.599612 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:44:35.599615 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:35.599628 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:44:35.599665 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:35.600026 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:35.600030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:35.600035 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 10:44:35.600174 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 10:44:35.600225 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":287,"threads_started":1}
I20260928 10:44:35.601676 14139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122c8c61c68b429aa1d101d7594320ca" candidate_uuid: "05f4c5ca75654865be60bbb84d926aa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f071a3ad8e473488f3219543bc7d8a" is_pre_election: true
I20260928 10:44:35.601678 13878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122c8c61c68b429aa1d101d7594320ca" candidate_uuid: "05f4c5ca75654865be60bbb84d926aa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccd16872ce04ac4a59937a43ae740b4" is_pre_election: true
I20260928 10:44:35.602164 13878 raft_consensus.cc:2454] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4c5ca75654865be60bbb84d926aa0 in term 0.
I20260928 10:44:35.602165 14139 raft_consensus.cc:2454] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4c5ca75654865be60bbb84d926aa0 in term 0.
I20260928 10:44:35.602787 13945 leader_election.cc:302] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4c5ca75654865be60bbb84d926aa0, 34f071a3ad8e473488f3219543bc7d8a; no voters:
I20260928 10:44:35.603132 14208 raft_consensus.cc:2781] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:35.603261 14208 raft_consensus.cc:482] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:35.603355 14208 raft_consensus.cc:3037] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:35.605593 14208 raft_consensus.cc:504] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.606150 14208 leader_election.cc:288] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [CANDIDATE]: Term 1 election: Requested vote from peers 34f071a3ad8e473488f3219543bc7d8a (127.8.218.195:33817), 5ccd16872ce04ac4a59937a43ae740b4 (127.8.218.193:34443)
I20260928 10:44:35.606655 14139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122c8c61c68b429aa1d101d7594320ca" candidate_uuid: "05f4c5ca75654865be60bbb84d926aa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f071a3ad8e473488f3219543bc7d8a"
I20260928 10:44:35.606750 13878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122c8c61c68b429aa1d101d7594320ca" candidate_uuid: "05f4c5ca75654865be60bbb84d926aa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccd16872ce04ac4a59937a43ae740b4"
I20260928 10:44:35.606837 14139 raft_consensus.cc:3037] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:35.606979 13878 raft_consensus.cc:3037] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:35.609123 14139 raft_consensus.cc:2454] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4c5ca75654865be60bbb84d926aa0 in term 1.
I20260928 10:44:35.609238 13878 raft_consensus.cc:2454] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4c5ca75654865be60bbb84d926aa0 in term 1.
I20260928 10:44:35.609548 13945 leader_election.cc:302] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4c5ca75654865be60bbb84d926aa0, 34f071a3ad8e473488f3219543bc7d8a; no voters:
I20260928 10:44:35.609819 14208 raft_consensus.cc:2781] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:35.610203 14208 raft_consensus.cc:686] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 1 LEADER]: Becoming Leader. State: Replica: 05f4c5ca75654865be60bbb84d926aa0, State: Running, Role: LEADER
I20260928 10:44:35.610657 14208 consensus_queue.cc:237] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } }
I20260928 10:44:35.615737 13737 catalog_manager.cc:5495] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4c5ca75654865be60bbb84d926aa0 (127.8.218.194). New cstate: current_term: 1 leader_uuid: "05f4c5ca75654865be60bbb84d926aa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4c5ca75654865be60bbb84d926aa0" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 } health_report { overall_health: UNKNOWN } } }
I20260928 10:44:35.634644 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.632326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.8.218.194:42481)
0928 10:44:35.632592 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.632599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.632646 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:44:35.632888 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.632893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.632975 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.633061 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.633992 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.633998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.634002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.634020 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.634051 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.634063 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.634165 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.634271 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.634328 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:44:35.644084 14139 raft_consensus.cc:1261] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 1 FOLLOWER]: Refusing update from remote peer 05f4c5ca75654865be60bbb84d926aa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:35.644073 13878 raft_consensus.cc:1261] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 1 FOLLOWER]: Refusing update from remote peer 05f4c5ca75654865be60bbb84d926aa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:35.645275 14208 consensus_queue.cc:1035] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f071a3ad8e473488f3219543bc7d8a" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:35.645800 14213 consensus_queue.cc:1035] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ccd16872ce04ac4a59937a43ae740b4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 34443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:35.659098 14216 mvcc.cc:205] Tried to move back new op lower bound from 7334265961024946176 to 7334265960904986624. Current Snapshot: MvccSnapshot[applied={T|T < 7334265961024946176}]
I20260928 10:44:35.680387 13737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:35.680732 13737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:35.684547 13737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e89da4638d44d3d9eba4debfc220bee: Sending DeleteTablet for 3 replicas of tablet 122c8c61c68b429aa1d101d7594320ca
I20260928 10:44:35.686053 14119 tablet_service.cc:1514] Processing DeleteTablet for tablet 122c8c61c68b429aa1d101d7594320ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:35 UTC) from {username='slave'} at 127.0.0.1:39996
I20260928 10:44:35.686100 13989 tablet_service.cc:1514] Processing DeleteTablet for tablet 122c8c61c68b429aa1d101d7594320ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:35 UTC) from {username='slave'} at 127.0.0.1:46060
I20260928 10:44:35.687788 14227 tablet_replica.cc:330] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: stopping tablet replica
I20260928 10:44:35.687716 13858 tablet_service.cc:1514] Processing DeleteTablet for tablet 122c8c61c68b429aa1d101d7594320ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:35 UTC) from {username='slave'} at 127.0.0.1:35246
I20260928 10:44:35.688445 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 13796
I20260928 10:44:35.688586 14227 raft_consensus.cc:2229] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:35.689234 14227 raft_consensus.cc:2258] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:35.690452 14228 tablet_replica.cc:330] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4: stopping tablet replica
I20260928 10:44:35.690944 14228 raft_consensus.cc:2229] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:35.691664 14226 tablet_replica.cc:330] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: stopping tablet replica
I20260928 10:44:35.692157 14226 raft_consensus.cc:2229] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:35.693805 14228 raft_consensus.cc:2258] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:35.694180 14227 ts_tablet_manager.cc:1905] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:35.694469 14226 raft_consensus.cc:2258] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:35.696136 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.690469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.8.218.193:34443)
0928 10:44:35.690758 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.690764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.693386 (+ 2622us) server_negotiation.cc:407] Connection header received
0928 10:44:35.693836 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.693843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.693966 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.694044 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.695294 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.695300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.695305 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.695327 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.695368 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.695385 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.695508 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.695752 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.695825 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260928 10:44:35.697427 13918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:35.698544 14226 ts_tablet_manager.cc:1905] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:35.703142 14227 ts_tablet_manager.cc:1918] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:35.703490 14227 log.cc:1196] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/122c8c61c68b429aa1d101d7594320ca
I20260928 10:44:35.704159 14227 ts_tablet_manager.cc:1939] T 122c8c61c68b429aa1d101d7594320ca P 34f071a3ad8e473488f3219543bc7d8a: Deleting consensus metadata
I20260928 10:44:35.706759 13724 catalog_manager.cc:4841] TS 34f071a3ad8e473488f3219543bc7d8a (127.8.218.195:33817): tablet 122c8c61c68b429aa1d101d7594320ca (table test-table [id=89a22040e49640829127b4c673eab9d4]) successfully deleted
I20260928 10:44:35.707290 14226 ts_tablet_manager.cc:1918] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:35.707580 14226 log.cc:1196] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/122c8c61c68b429aa1d101d7594320ca
I20260928 10:44:35.708362 14226 ts_tablet_manager.cc:1939] T 122c8c61c68b429aa1d101d7594320ca P 05f4c5ca75654865be60bbb84d926aa0: Deleting consensus metadata
I20260928 10:44:35.710795 13723 catalog_manager.cc:4841] TS 05f4c5ca75654865be60bbb84d926aa0 (127.8.218.194:42481): tablet 122c8c61c68b429aa1d101d7594320ca (table test-table [id=89a22040e49640829127b4c673eab9d4]) successfully deleted
I20260928 10:44:35.705293 14228 ts_tablet_manager.cc:1905] T 122c8c61c68b429aa1d101d7594320ca P 5ccd16872ce04ac4a59937a43ae740b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:35.794252 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 13796
I20260928 10:44:35.816524 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 13927
W20260928 10:44:35.816550 13724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.218.193:34443 (error 108)
I20260928 10:44:35.819739 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.819238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.0.0.1:35260)
0928 10:44:35.819422 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.819551 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 10:44:35.820112 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.817381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.8.218.194:42481)
0928 10:44:35.817647 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.817653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.817856 (+ 203us) server_negotiation.cc:407] Connection header received
0928 10:44:35.818077 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.818083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.818158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.818272 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.819413 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.819420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.819425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.819445 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.819485 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.819500 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.819620 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.819762 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.819821 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 10:44:35.820175 13724 catalog_manager.cc:4584] TS 5ccd16872ce04ac4a59937a43ae740b4 (127.8.218.193:34443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 122c8c61c68b429aa1d101d7594320ca: Network error: Client connection negotiation failed: client connection to 127.8.218.193:34443: connect: Connection refused (error 111)
I20260928 10:44:35.820885 14049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:35.874002 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.873325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.0.0.1:35276)
0928 10:44:35.873661 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.873767 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 10:44:35.875154 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.874736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.0.0.1:35286)
0928 10:44:35.874930 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.875021 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 10:44:35.930327 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 13927
I20260928 10:44:35.935663 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.935208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.0.0.1:35288)
0928 10:44:35.935401 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.935514 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 10:44:35.936686 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.936341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.0.0.1:35296)
0928 10:44:35.936480 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:35.936553 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 10:44:35.953686 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 14057
I20260928 10:44:35.956447 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:35.954456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.8.218.195:33817)
0928 10:44:35.954626 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:35.954631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:35.954649 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:44:35.954822 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:35.954827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:35.954904 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:35.954991 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:35.955933 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:35.955938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:35.955941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:35.955956 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:35.955987 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:35.955998 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:35.956099 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:35.956182 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 10:44:35.956230 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:44:35.957051 14179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:36.017488 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:36.016976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.0.0.1:35308)
0928 10:44:36.017203 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:36.017308 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 10:44:36.018528 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:36.018119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:34443 (local address 127.0.0.1:35314)
0928 10:44:36.018330 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:36.018403 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 10:44:36.045348 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 14057
I20260928 10:44:36.058475 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 13706
I20260928 10:44:36.059571 13767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:36.166309 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 13706
2026-09-28T10:44:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2732 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 10:44:36.204208 9067 mini_oidc.cc:152] Starting JWKS server
I20260928 10:44:36.206919 9067 webserver.cc:460] Webserver started at https://127.0.0.1:42531/ using document root <none> and password file <none>
I20260928 10:44:36.207231 9067 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:44:36.207996 9067 webserver.cc:460] Webserver started at http://127.0.0.1:43491/ using document root <none> and password file <none>
2026-09-28T10:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:36Z Disabled control of system clock
I20260928 10:44:36.235551 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:42899
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:42899
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/testchainca.cert with env {}
W20260928 10:44:36.577641 14240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:36.577950 14240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:36.578047 14240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:36.585544 14240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:36.585675 14240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:36.585711 14240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:36.585759 14240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:36.585803 14240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:44:36.585866 14240 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:44:36.585999 14240 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42531/jwks.json
I20260928 10:44:36.596531 14240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:42899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:36.597880 14240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:36.599152 14240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:36.611006 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:36.611034 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:36.610965 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:36.611909 14240 server_base.cc:1034] running on GCE node
I20260928 10:44:36.612890 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:36.614879 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:36.616160 14240 hybrid_clock.cc:648] HybridClock initialized: now 1790592276616089 us; error 56 us; skew 500 ppm
I20260928 10:44:36.616600 14240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:44:36.619524 14240 webserver.cc:460] Webserver started at http://127.8.218.254:43033/ using document root <none> and password file <none>
I20260928 10:44:36.620378 14240 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:36.620519 14240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:36.620831 14240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:36.624076 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "55689bc7d87c4bd18788cea976400ee6"
format_stamp: "Formatted at 2026-09-28 10:44:36 on dist-test-slave-mn38"
server_key: "4f694d7de6a3a001a98010063b89502e"
server_key_iv: "ab223344402e4a5783ebe14d209faddc"
server_key_version: "encryptionkey@0"
I20260928 10:44:36.624960 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "55689bc7d87c4bd18788cea976400ee6"
format_stamp: "Formatted at 2026-09-28 10:44:36 on dist-test-slave-mn38"
server_key: "4f694d7de6a3a001a98010063b89502e"
server_key_iv: "ab223344402e4a5783ebe14d209faddc"
server_key_version: "encryptionkey@0"
I20260928 10:44:36.630630 14240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 10:44:36.634737 14254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:36.636358 14240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:44:36.636566 14240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "55689bc7d87c4bd18788cea976400ee6"
format_stamp: "Formatted at 2026-09-28 10:44:36 on dist-test-slave-mn38"
server_key: "4f694d7de6a3a001a98010063b89502e"
server_key_iv: "ab223344402e4a5783ebe14d209faddc"
server_key_version: "encryptionkey@0"
I20260928 10:44:36.636803 14240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:36.665843 14240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:36.666735 14240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:36.667032 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:36.686605 14240 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:42899
I20260928 10:44:36.686636 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:42899 every 8 connection(s)
I20260928 10:44:36.688063 14240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:36.692258 14306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:36.696589 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 14240
I20260928 10:44:36.696801 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:36.697608 9067 external_mini_cluster.cc:1428] Setting key 65436757cc898a2b83aa3a2c11a37a04
I20260928 10:44:36.706178 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6: Bootstrap starting.
I20260928 10:44:36.709642 14306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:36.710837 14306 log.cc:824] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:36.711381 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:36.699697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.8.218.254:42899)
0928 10:44:36.700670 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:36.700681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:36.700723 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:44:36.702419 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:36.702449 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:36.703042 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:36.703467 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:36.704082 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:36.705370 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:36.706783 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:36.707487 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:36.707700 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:36.708894 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:36.708902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:36.708917 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:36.709561 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:36.709670 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:36.709693 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:36.709803 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:36.710028 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 10:44:36.710355 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":292,"threads_started":1}
I20260928 10:44:36.714880 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6: No bootstrap required, opened a new log
I20260928 10:44:36.716733 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:44:36.724809 14306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55689bc7d87c4bd18788cea976400ee6" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42899 } }
I20260928 10:44:36.725271 14306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:36.725353 14306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55689bc7d87c4bd18788cea976400ee6, State: Initialized, Role: FOLLOWER
I20260928 10:44:36.726003 14306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55689bc7d87c4bd18788cea976400ee6" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42899 } }
I20260928 10:44:36.726219 14306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:36.726318 14306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:36.726446 14306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:36.729081 14306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55689bc7d87c4bd18788cea976400ee6" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42899 } }
I20260928 10:44:36.729630 14306 leader_election.cc:302] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55689bc7d87c4bd18788cea976400ee6; no voters:
I20260928 10:44:36.730170 14306 leader_election.cc:288] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:36.730332 14311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:36.731024 14311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [term 1 LEADER]: Becoming Leader. State: Replica: 55689bc7d87c4bd18788cea976400ee6, State: Running, Role: LEADER
I20260928 10:44:36.731592 14311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55689bc7d87c4bd18788cea976400ee6" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42899 } }
I20260928 10:44:36.732300 14306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:36.734542 14313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55689bc7d87c4bd18788cea976400ee6. Latest consensus state: current_term: 1 leader_uuid: "55689bc7d87c4bd18788cea976400ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55689bc7d87c4bd18788cea976400ee6" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42899 } } }
I20260928 10:44:36.734604 14312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55689bc7d87c4bd18788cea976400ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55689bc7d87c4bd18788cea976400ee6" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42899 } } }
I20260928 10:44:36.734946 14312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:36.734946 14313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:36.736930 14317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:36.742415 14317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:36.751518 14317 catalog_manager.cc:1347] Generated new cluster ID: d4104800faea4f30ad2a5369a8ad9ee5
I20260928 10:44:36.751670 14317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:36.794313 14317 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:36.795655 14317 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:36.804929 14317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55689bc7d87c4bd18788cea976400ee6: Generated new TSK 0
I20260928 10:44:36.805941 14317 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 10:44:36.857240 14336 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 10:44:36.857430 14336 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:44:36.858423 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:36.828670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.8.218.254:42899)
0928 10:44:36.828894 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:36.828905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:36.829464 (+ 559us) server_negotiation.cc:407] Connection header received
0928 10:44:36.829846 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:36.829851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:36.829923 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:36.830079 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:44:36.830687 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:36.831431 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:36.833706 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:36.834233 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:36.834349 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:36.835093 (+ 744us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:44:36.857786 (+ 22693us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 10:44:36.858127 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 10:44:36.858577 14309 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 10:44:36.860430 14335 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:36.828561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42899 (local address 127.0.0.1:48674)
0928 10:44:36.829307 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:36.829334 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:36.829577 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:36.830127 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:36.830133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:36.830148 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:44:36.830530 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:36.830542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:36.831583 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:36.831589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:36.833504 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:36.833524 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:36.834416 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:36.834422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:36.834635 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:36.834736 (+ 101us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:44:36.858112 (+ 23376us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 10:44:36.858501 (+ 389us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.218.254:42899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":546,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.218.254:42899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 10:44:36.862471 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 14240
I20260928 10:44:36.863821 14301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:36.961757 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 14240
2026-09-28T10:44:36Z chronyd exiting
I20260928 10:44:37.010377 9067 test_util.cc:175] -----------------------------------------------
I20260928 10:44:37.010522 9067 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790592230404718-9067-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (810 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 10:44:37.014556 9067 mini_oidc.cc:152] Starting JWKS server
I20260928 10:44:37.016489 9067 webserver.cc:460] Webserver started at https://127.0.0.1:38193/ using document root <none> and password file <none>
I20260928 10:44:37.016750 9067 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:44:37.017421 9067 webserver.cc:460] Webserver started at http://127.0.0.1:34493/ using document root <none> and password file <none>
2026-09-28T10:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:37Z Disabled control of system clock
I20260928 10:44:37.044446 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45175
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:45175
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38193/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/testchainca.cert with env {}
W20260928 10:44:37.387476 14343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:37.387794 14343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:37.387882 14343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:37.395437 14343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:37.395565 14343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:37.395601 14343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:37.395663 14343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:37.395709 14343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:44:37.395769 14343 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:44:37.395809 14343 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38193/jwks.json
I20260928 10:44:37.406107 14343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:45175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38193/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:37.407438 14343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:37.408946 14343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:37.422129 14351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:37.422168 14349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:37.422363 14348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:37.423338 14343 server_base.cc:1034] running on GCE node
I20260928 10:44:37.424187 14343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:37.426123 14343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:37.427402 14343 hybrid_clock.cc:648] HybridClock initialized: now 1790592277427334 us; error 66 us; skew 500 ppm
I20260928 10:44:37.427896 14343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:44:37.430845 14343 webserver.cc:460] Webserver started at http://127.8.218.254:37915/ using document root <none> and password file <none>
I20260928 10:44:37.431668 14343 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:37.431797 14343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:37.432139 14343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:37.435400 14343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "12a0f524a3524821a4b2b1b556e988a8"
format_stamp: "Formatted at 2026-09-28 10:44:37 on dist-test-slave-mn38"
server_key: "42b1af09d4807ae0480f73519a211f31"
server_key_iv: "fb9cc8981063445d4048282cbcfdd6cb"
server_key_version: "encryptionkey@0"
I20260928 10:44:37.436283 14343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "12a0f524a3524821a4b2b1b556e988a8"
format_stamp: "Formatted at 2026-09-28 10:44:37 on dist-test-slave-mn38"
server_key: "42b1af09d4807ae0480f73519a211f31"
server_key_iv: "fb9cc8981063445d4048282cbcfdd6cb"
server_key_version: "encryptionkey@0"
I20260928 10:44:37.442233 14343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:44:37.446237 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:37.447891 14343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:44:37.448117 14343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "12a0f524a3524821a4b2b1b556e988a8"
format_stamp: "Formatted at 2026-09-28 10:44:37 on dist-test-slave-mn38"
server_key: "42b1af09d4807ae0480f73519a211f31"
server_key_iv: "fb9cc8981063445d4048282cbcfdd6cb"
server_key_version: "encryptionkey@0"
I20260928 10:44:37.448356 14343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:37.482056 14343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:37.483001 14343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:37.483309 14343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:37.501488 14343 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:45175
I20260928 10:44:37.501498 14408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:45175 every 8 connection(s)
I20260928 10:44:37.502889 14343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:37.505420 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 14343
I20260928 10:44:37.505641 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:37.506250 9067 external_mini_cluster.cc:1428] Setting key 689b8523feaa50ca6225597bb00b351b
I20260928 10:44:37.507087 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:37.520591 14409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8: Bootstrap starting.
I20260928 10:44:37.521490 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:37.508246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.8.218.254:45175)
0928 10:44:37.509500 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:37.509519 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:37.509569 (+ 50us) server_negotiation.cc:407] Connection header received
0928 10:44:37.512114 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:37.512150 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:37.512759 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:37.513220 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:37.513945 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:37.515339 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:37.516706 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:37.517483 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:37.517667 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:37.518837 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:37.518846 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:37.518863 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:37.519463 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:37.519547 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:37.519570 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:37.519670 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:37.519901 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 10:44:37.520234 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":428,"threads_started":1}
I20260928 10:44:37.523989 14409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:37.525151 14409 log.cc:824] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:37.525394 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:44:37.528177 14409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8: No bootstrap required, opened a new log
I20260928 10:44:37.536108 14409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0f524a3524821a4b2b1b556e988a8" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45175 } }
I20260928 10:44:37.536542 14409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:37.536654 14409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a0f524a3524821a4b2b1b556e988a8, State: Initialized, Role: FOLLOWER
I20260928 10:44:37.537231 14409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0f524a3524821a4b2b1b556e988a8" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45175 } }
I20260928 10:44:37.537461 14409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:37.537596 14409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:37.537755 14409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:37.540081 14409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0f524a3524821a4b2b1b556e988a8" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45175 } }
I20260928 10:44:37.540570 14409 leader_election.cc:302] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a0f524a3524821a4b2b1b556e988a8; no voters:
I20260928 10:44:37.541122 14409 leader_election.cc:288] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:37.541247 14414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:37.542305 14414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [term 1 LEADER]: Becoming Leader. State: Replica: 12a0f524a3524821a4b2b1b556e988a8, State: Running, Role: LEADER
I20260928 10:44:37.542919 14414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0f524a3524821a4b2b1b556e988a8" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45175 } }
I20260928 10:44:37.543438 14409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:37.545608 14415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12a0f524a3524821a4b2b1b556e988a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0f524a3524821a4b2b1b556e988a8" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45175 } } }
I20260928 10:44:37.545931 14415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:37.546339 14416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a0f524a3524821a4b2b1b556e988a8. Latest consensus state: current_term: 1 leader_uuid: "12a0f524a3524821a4b2b1b556e988a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a0f524a3524821a4b2b1b556e988a8" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45175 } } }
I20260928 10:44:37.546644 14416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:37.546761 14423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:37.553161 14423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:37.566942 14423 catalog_manager.cc:1347] Generated new cluster ID: 8862491b9d5248fdb0327bf376a2172e
I20260928 10:44:37.567067 14423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:37.589098 14423 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:37.590636 14423 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:37.597965 14423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12a0f524a3524821a4b2b1b556e988a8: Generated new TSK 0
I20260928 10:44:37.598896 14423 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 10:44:37.643266 14439 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 10:44:37.643456 14439 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:44:37.644622 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:37.617498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.8.218.254:45175)
0928 10:44:37.617660 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:37.617667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:37.618158 (+ 491us) server_negotiation.cc:407] Connection header received
0928 10:44:37.618409 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:37.618413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:37.618490 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:37.618627 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:44:37.619112 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:37.619873 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:37.621289 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:37.621811 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:37.621940 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:37.622474 (+ 534us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:44:37.643915 (+ 21441us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38193/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 10:44:37.644281 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38193/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 10:44:37.644758 14412 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38193/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 10:44:37.645015 14438 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:37.617413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45175 (local address 127.0.0.1:44308)
0928 10:44:37.618057 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:37.618071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:37.618217 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:37.618656 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:37.618661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:37.618672 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:44:37.618942 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:37.618953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:37.620023 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:37.620029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:37.621132 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:37.621145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:37.622013 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:37.622020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:37.622176 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:37.622214 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:44:37.644168 (+ 21954us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38193/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 10:44:37.644472 (+ 304us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.218.254:45175: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38193/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":327,"threads_started":1}
I20260928 10:44:37.646096 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 14343
I20260928 10:44:37.647362 14404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:37.741106 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 14343
2026-09-28T10:44:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (779 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 10:44:37.794113 9067 mini_oidc.cc:152] Starting JWKS server
I20260928 10:44:37.796082 9067 webserver.cc:460] Webserver started at https://127.0.0.1:37247/ using document root <none> and password file <none>
I20260928 10:44:37.796324 9067 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 10:44:37.797019 9067 webserver.cc:460] Webserver started at http://127.0.0.1:37665/ using document root <none> and password file <none>
2026-09-28T10:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:37Z Disabled control of system clock
I20260928 10:44:37.824357 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:33493
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:33493
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37247/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 10:44:38.167917 14446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:38.168259 14446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:38.168376 14446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:38.176108 14446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:38.176256 14446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:38.176312 14446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:38.176389 14446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:38.176451 14446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:44:38.176537 14446 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:44:38.176584 14446 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37247/jwks.json
I20260928 10:44:38.186234 14446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:33493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37247/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:38.187577 14446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:38.189488 14446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:38.202347 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:38.202382 14454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:38.202412 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:38.203277 14446 server_base.cc:1034] running on GCE node
I20260928 10:44:38.204214 14446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:38.206343 14446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:38.207664 14446 hybrid_clock.cc:648] HybridClock initialized: now 1790592278207541 us; error 123 us; skew 500 ppm
I20260928 10:44:38.208217 14446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:44:38.210908 14446 webserver.cc:460] Webserver started at http://127.8.218.254:36573/ using document root <none> and password file <none>
I20260928 10:44:38.211704 14446 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:38.211827 14446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:38.212172 14446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:38.215682 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "a3281f58267b46d0b429bbb948c0b069"
format_stamp: "Formatted at 2026-09-28 10:44:38 on dist-test-slave-mn38"
server_key: "01c927d10945d00c0bc76d40cd82c054"
server_key_iv: "d62b00d20d77d898978f619de112ab14"
server_key_version: "encryptionkey@0"
I20260928 10:44:38.216610 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "a3281f58267b46d0b429bbb948c0b069"
format_stamp: "Formatted at 2026-09-28 10:44:38 on dist-test-slave-mn38"
server_key: "01c927d10945d00c0bc76d40cd82c054"
server_key_iv: "d62b00d20d77d898978f619de112ab14"
server_key_version: "encryptionkey@0"
I20260928 10:44:38.222216 14446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 10:44:38.225894 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:38.227515 14446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:44:38.227717 14446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "a3281f58267b46d0b429bbb948c0b069"
format_stamp: "Formatted at 2026-09-28 10:44:38 on dist-test-slave-mn38"
server_key: "01c927d10945d00c0bc76d40cd82c054"
server_key_iv: "d62b00d20d77d898978f619de112ab14"
server_key_version: "encryptionkey@0"
I20260928 10:44:38.227949 14446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:38.254907 14446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:38.255822 14446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:38.256135 14446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:38.274859 14446 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:33493
I20260928 10:44:38.274972 14511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:33493 every 8 connection(s)
I20260928 10:44:38.276382 14446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:38.280686 14512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:38.285709 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 14446
I20260928 10:44:38.285926 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:38.286681 9067 external_mini_cluster.cc:1428] Setting key 2be30dfb236ffa2621ed476ae7a8ea7e
I20260928 10:44:38.293074 14512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069: Bootstrap starting.
I20260928 10:44:38.297106 14512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:38.298513 14512 log.cc:824] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:38.302403 14512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069: No bootstrap required, opened a new log
I20260928 10:44:38.303160 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:38.288802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.8.218.254:33493)
0928 10:44:38.290028 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:38.290045 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:38.290097 (+ 52us) server_negotiation.cc:407] Connection header received
0928 10:44:38.291950 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:38.291977 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:38.292656 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:38.293114 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:38.293820 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:38.295213 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:38.297520 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:38.298327 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:38.298558 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:38.299901 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:38.299913 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:38.299940 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:38.300772 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:38.300904 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:38.300938 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:38.301082 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:38.301342 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 10:44:38.301756 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":403,"threads_started":1}
I20260928 10:44:38.307272 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:44:38.311655 14512 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3281f58267b46d0b429bbb948c0b069" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33493 } }
I20260928 10:44:38.312044 14512 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:38.312119 14512 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3281f58267b46d0b429bbb948c0b069, State: Initialized, Role: FOLLOWER
I20260928 10:44:38.312649 14512 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3281f58267b46d0b429bbb948c0b069" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33493 } }
I20260928 10:44:38.312865 14512 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:38.312966 14512 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:38.313097 14512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:38.315342 14512 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3281f58267b46d0b429bbb948c0b069" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33493 } }
I20260928 10:44:38.315958 14512 leader_election.cc:302] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3281f58267b46d0b429bbb948c0b069; no voters:
I20260928 10:44:38.316571 14512 leader_election.cc:288] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:38.316722 14517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:38.317494 14517 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [term 1 LEADER]: Becoming Leader. State: Replica: a3281f58267b46d0b429bbb948c0b069, State: Running, Role: LEADER
I20260928 10:44:38.318106 14517 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3281f58267b46d0b429bbb948c0b069" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33493 } }
I20260928 10:44:38.318714 14512 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:38.321331 14519 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3281f58267b46d0b429bbb948c0b069. Latest consensus state: current_term: 1 leader_uuid: "a3281f58267b46d0b429bbb948c0b069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3281f58267b46d0b429bbb948c0b069" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33493 } } }
I20260928 10:44:38.321892 14519 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:38.320763 14518 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3281f58267b46d0b429bbb948c0b069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3281f58267b46d0b429bbb948c0b069" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33493 } } }
I20260928 10:44:38.322732 14518 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:38.324092 14524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:38.330240 14524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:38.339710 14524 catalog_manager.cc:1347] Generated new cluster ID: b4a03c812dc4438fb32181fc6497a04e
I20260928 10:44:38.339836 14524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:38.349550 14524 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:38.350944 14524 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:38.359627 14524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3281f58267b46d0b429bbb948c0b069: Generated new TSK 0
I20260928 10:44:38.360739 14524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:44:38.391330 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:38.382164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.8.218.254:33493)
0928 10:44:38.382413 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:38.382420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:38.382822 (+ 402us) server_negotiation.cc:407] Connection header received
0928 10:44:38.383046 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:38.383052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:38.383155 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:38.383347 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:44:38.383760 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:38.384897 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:38.386483 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:38.387199 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:38.387395 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:38.387985 (+ 590us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:44:38.390639 (+ 2654us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37247/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 10:44:38.391010 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36484: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37247/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 10:44:38.391469 14515 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36484: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37247/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 10:44:38.393006 14541 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:38.382004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:33493 (local address 127.0.0.1:36484)
0928 10:44:38.382709 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:38.382727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:38.382880 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:38.383350 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:38.383354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:38.383365 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:44:38.383613 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:38.383621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:38.385069 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:38.385073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:38.386299 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:38.386318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:38.387457 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:38.387462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:38.387611 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:38.387652 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:44:38.390913 (+ 3261us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37247/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 10:44:38.392555 (+ 1642us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.218.254:33493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37247/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":1289,"thread_start_us":406,"threads_started":1}
I20260928 10:44:38.394423 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 14446
I20260928 10:44:38.395889 14507 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 10:44:38.402946 14542 webserver.cc:533] Webserver: sslize failed:
E20260928 10:44:38.403055 14542 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 10:44:38.488584 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 14446
2026-09-28T10:44:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T10:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:38Z Disabled control of system clock
I20260928 10:44:38.569097 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:46303
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:46303
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/testchainca.cert with env {}
W20260928 10:44:38.910223 14547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:38.910538 14547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:38.910676 14547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:38.918651 14547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:38.918798 14547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:38.918850 14547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:38.918937 14547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:38.918994 14547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 10:44:38.919118 14547 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 10:44:38.919170 14547 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 10:44:38.930151 14547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:46303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:46303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:38.931566 14547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:38.933306 14547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:38.945365 14552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:38.945472 14553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:38.945722 14555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:38.946506 14547 server_base.cc:1034] running on GCE node
I20260928 10:44:38.947372 14547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:38.948851 14547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:38.950089 14547 hybrid_clock.cc:648] HybridClock initialized: now 1790592278950044 us; error 35 us; skew 500 ppm
I20260928 10:44:38.950474 14547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:44:38.958348 14547 webserver.cc:460] Webserver started at http://127.8.218.254:36741/ using document root <none> and password file <none>
I20260928 10:44:38.959172 14547 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:38.959295 14547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:38.959640 14547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:38.962759 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "4ff15b6e4c1e4d98b260381c5a32f434"
format_stamp: "Formatted at 2026-09-28 10:44:38 on dist-test-slave-mn38"
server_key: "6f6a5514cffd4bea03ca053fa1b85371"
server_key_iv: "85f0cfd397654e5d2968b2cfba5b75d9"
server_key_version: "encryptionkey@0"
I20260928 10:44:38.963567 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "4ff15b6e4c1e4d98b260381c5a32f434"
format_stamp: "Formatted at 2026-09-28 10:44:38 on dist-test-slave-mn38"
server_key: "6f6a5514cffd4bea03ca053fa1b85371"
server_key_iv: "85f0cfd397654e5d2968b2cfba5b75d9"
server_key_version: "encryptionkey@0"
I20260928 10:44:38.969106 14547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:44:38.972978 14561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:38.974601 14547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:44:38.974790 14547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "4ff15b6e4c1e4d98b260381c5a32f434"
format_stamp: "Formatted at 2026-09-28 10:44:38 on dist-test-slave-mn38"
server_key: "6f6a5514cffd4bea03ca053fa1b85371"
server_key_iv: "85f0cfd397654e5d2968b2cfba5b75d9"
server_key_version: "encryptionkey@0"
I20260928 10:44:38.975009 14547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:39.000936 14547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:39.001981 14547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:39.002312 14547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:39.021840 14547 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:46303
I20260928 10:44:39.021867 14612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:46303 every 8 connection(s)
I20260928 10:44:39.023228 14547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:39.027333 14613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:39.030681 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 14547
I20260928 10:44:39.030907 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:39.031674 9067 external_mini_cluster.cc:1428] Setting key 45407f3ee5d761c029e02f158b92795b
I20260928 10:44:39.040294 14613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434: Bootstrap starting.
I20260928 10:44:39.043586 14613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:39.044715 14613 log.cc:824] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:39.045576 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:39.033746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.8.218.254:46303)
0928 10:44:39.034855 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:39.034868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:39.034912 (+ 44us) server_negotiation.cc:407] Connection header received
0928 10:44:39.036603 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:39.036629 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:39.037179 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:39.037628 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:39.038250 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:39.039566 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:39.040933 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:39.041730 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:39.041916 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:39.043082 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:39.043090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:39.043105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:44:39.043676 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:39.043792 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:44:39.043816 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:39.043934 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:39.044149 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 10:44:39.044480 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":344,"threads_started":1}
I20260928 10:44:39.047935 14613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434: No bootstrap required, opened a new log
I20260928 10:44:39.049681 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:44:39.057011 14613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff15b6e4c1e4d98b260381c5a32f434" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46303 } }
I20260928 10:44:39.057412 14613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:39.057528 14613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff15b6e4c1e4d98b260381c5a32f434, State: Initialized, Role: FOLLOWER
I20260928 10:44:39.058068 14613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff15b6e4c1e4d98b260381c5a32f434" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46303 } }
I20260928 10:44:39.058269 14613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:39.058365 14613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:39.058492 14613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:39.060562 14613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff15b6e4c1e4d98b260381c5a32f434" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46303 } }
I20260928 10:44:39.061069 14613 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ff15b6e4c1e4d98b260381c5a32f434; no voters:
I20260928 10:44:39.061694 14613 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:39.061825 14618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:39.062505 14618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [term 1 LEADER]: Becoming Leader. State: Replica: 4ff15b6e4c1e4d98b260381c5a32f434, State: Running, Role: LEADER
I20260928 10:44:39.063252 14618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff15b6e4c1e4d98b260381c5a32f434" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46303 } }
I20260928 10:44:39.064036 14613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:39.065948 14619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ff15b6e4c1e4d98b260381c5a32f434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff15b6e4c1e4d98b260381c5a32f434" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46303 } } }
I20260928 10:44:39.066908 14619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:39.066546 14620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff15b6e4c1e4d98b260381c5a32f434. Latest consensus state: current_term: 1 leader_uuid: "4ff15b6e4c1e4d98b260381c5a32f434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff15b6e4c1e4d98b260381c5a32f434" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 46303 } } }
I20260928 10:44:39.067265 14620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:39.068543 14627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:39.074252 14627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:39.084249 14627 catalog_manager.cc:1347] Generated new cluster ID: 3bbc80d19ae04af4baa2e7d79700bec8
I20260928 10:44:39.084373 14627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:39.108739 14627 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:39.110431 14627 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:39.123018 14627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ff15b6e4c1e4d98b260381c5a32f434: Generated new TSK 0
I20260928 10:44:39.124145 14627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:44:39.615005 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:39.147193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.8.218.254:46303)
0928 10:44:39.147391 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:39.147397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:39.147733 (+ 336us) server_negotiation.cc:407] Connection header received
0928 10:44:39.147950 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:39.147954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:39.148035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:39.148166 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0928 10:44:39.148689 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:39.149535 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:39.151649 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:39.152385 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:39.152529 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:39.153070 (+ 541us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 10:44:39.614462 (+461392us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 10:44:39.614786 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51444: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 10:44:39.615136 14616 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51444: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 10:44:39.615339 14642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:39.146976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:46303 (local address 127.0.0.1:51444)
0928 10:44:39.147603 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:39.147624 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:39.147782 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:39.148203 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:39.148209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:39.148222 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 10:44:39.148533 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:39.148544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:39.149689 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:39.149694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:39.151452 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:39.151471 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:39.152566 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:39.152573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:39.152759 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:39.152808 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 10:44:39.614769 (+461961us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 10:44:39.615008 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.218.254:46303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":298,"threads_started":1}
I20260928 10:44:39.616516 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 14547
I20260928 10:44:39.617638 14608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:39.719913 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 14547
2026-09-28T10:44:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1212 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (426 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1426 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:41 dist-test-slave-mn38 krb5kdc[14658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:41 dist-test-slave-mn38 krb5kdc[14658](info): set up 2 sockets
Sep 28 10:44:41 dist-test-slave-mn38 krb5kdc[14658](info): commencing operation
krb5kdc: starting...
W20260928 10:44:42.583030 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:44:42 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:44:42.794597 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36817
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:43.134702 14671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:43.135005 14671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:43.135100 14671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:43.142619 14671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:43.142778 14671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:43.142827 14671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:43.142885 14671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:43.142930 14671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:43.153558 14671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:36817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:43.154965 14671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:43.156419 14671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:43.170450 14679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:43.170476 14676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:43.170498 14677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:43.171379 14671 server_base.cc:1034] running on GCE node
I20260928 10:44:43.172241 14671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:43.173980 14671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:43.175266 14671 hybrid_clock.cc:648] HybridClock initialized: now 1790592283175169 us; error 71 us; skew 500 ppm
I20260928 10:44:43.175647 14671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:43 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592283, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:43.179140 14671 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:44:43.180835 14671 webserver.cc:460] Webserver started at http://127.8.218.254:35223/ using document root <none> and password file <none>
I20260928 10:44:43.181622 14671 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:43.181751 14671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:43.182065 14671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:43.185281 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "a7de29c4c29d45bd96ace69cf325696f"
format_stamp: "Formatted at 2026-09-28 10:44:43 on dist-test-slave-mn38"
server_key: "c4bfc2204774e879070bae5ed89bb8bc"
server_key_iv: "2e36ff0734fc16fa955ff439d65e93fd"
server_key_version: "encryptionkey@0"
I20260928 10:44:43.186152 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "a7de29c4c29d45bd96ace69cf325696f"
format_stamp: "Formatted at 2026-09-28 10:44:43 on dist-test-slave-mn38"
server_key: "c4bfc2204774e879070bae5ed89bb8bc"
server_key_iv: "2e36ff0734fc16fa955ff439d65e93fd"
server_key_version: "encryptionkey@0"
I20260928 10:44:43.191879 14671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:44:43.196209 14686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:43.198025 14671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:43.198239 14671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "a7de29c4c29d45bd96ace69cf325696f"
format_stamp: "Formatted at 2026-09-28 10:44:43 on dist-test-slave-mn38"
server_key: "c4bfc2204774e879070bae5ed89bb8bc"
server_key_iv: "2e36ff0734fc16fa955ff439d65e93fd"
server_key_version: "encryptionkey@0"
I20260928 10:44:43.198458 14671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:43.219539 14671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:43.224279 14671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:43.224560 14671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:43.243901 14671 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:36817
I20260928 10:44:43.243933 14737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:36817 every 8 connection(s)
I20260928 10:44:43.245994 14671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:43.250442 14738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:43.256172 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 14671
I20260928 10:44:43.256400 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:43.257000 9067 external_mini_cluster.cc:1428] Setting key ee95e80a6d5ec2532d218474f2b19296
I20260928 10:44:43.262396 14738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f: Bootstrap starting.
I20260928 10:44:43.266461 14738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:43.267581 14738 log.cc:824] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:43.270820 14738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f: No bootstrap required, opened a new log
Sep 28 10:44:43 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:43.281042 14738 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7de29c4c29d45bd96ace69cf325696f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36817 } }
I20260928 10:44:43.281672 14738 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:43.281788 14738 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7de29c4c29d45bd96ace69cf325696f, State: Initialized, Role: FOLLOWER
I20260928 10:44:43.282599 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:43.259214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.8.218.254:36817)
0928 10:44:43.260180 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:43.260195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:43.260244 (+ 49us) server_negotiation.cc:407] Connection header received
0928 10:44:43.262157 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:43.262186 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:43.262711 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:43.263099 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:43.264963 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:43.266477 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:43.267880 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:43.268838 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:43.268994 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:43.272849 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:43.272861 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:43.272885 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:43.272943 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:43.277148 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:43.278104 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:43.278114 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:43.278124 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:43.278371 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:43.278815 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:43.278821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:43.278825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:43.279740 (+ 915us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:43.280056 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:43.280587 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 10:44:43.281089 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":460,"threads_started":1}
I20260928 10:44:43.282541 14738 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7de29c4c29d45bd96ace69cf325696f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36817 } }
I20260928 10:44:43.282850 14738 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:43.283036 14738 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:43.283226 14738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:43.286427 14738 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7de29c4c29d45bd96ace69cf325696f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36817 } }
I20260928 10:44:43.287006 14738 leader_election.cc:302] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7de29c4c29d45bd96ace69cf325696f; no voters:
I20260928 10:44:43.287606 14738 leader_election.cc:288] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:43.287766 14743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:43.288475 14743 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [term 1 LEADER]: Becoming Leader. State: Replica: a7de29c4c29d45bd96ace69cf325696f, State: Running, Role: LEADER
I20260928 10:44:43.289054 14743 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7de29c4c29d45bd96ace69cf325696f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36817 } }
I20260928 10:44:43.289968 14738 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:43.292078 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7de29c4c29d45bd96ace69cf325696f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7de29c4c29d45bd96ace69cf325696f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36817 } } }
I20260928 10:44:43.292219 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7de29c4c29d45bd96ace69cf325696f. Latest consensus state: current_term: 1 leader_uuid: "a7de29c4c29d45bd96ace69cf325696f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7de29c4c29d45bd96ace69cf325696f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36817 } } }
I20260928 10:44:43.292585 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:43.292752 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:43.296473 14750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:43.302600 14750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:43.311715 14750 catalog_manager.cc:1347] Generated new cluster ID: 5a58dea7ed9a4363b0f7ee630d902a17
I20260928 10:44:43.311846 14750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:43.329387 14750 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:43.330862 14750 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:43.337916 14750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7de29c4c29d45bd96ace69cf325696f: Generated new TSK 0
I20260928 10:44:43.338871 14750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:44:43.399214 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36817
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:43.762054 14764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:43.762355 14764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:43.762472 14764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:43.770372 14764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:43.770576 14764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:43.770750 14764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:43.783303 14764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:43.784767 14764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:43.786387 14764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:43.802590 14772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:43.802657 14769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:43.802748 14770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:43.804943 14764 server_base.cc:1034] running on GCE node
I20260928 10:44:43.806041 14764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:43.807328 14764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:43.808641 14764 hybrid_clock.cc:648] HybridClock initialized: now 1790592283808559 us; error 71 us; skew 500 ppm
I20260928 10:44:43.809059 14764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:43 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592283, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:43.812898 14764 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:44:43.814738 14764 webserver.cc:460] Webserver started at http://127.8.218.193:41861/ using document root <none> and password file <none>
I20260928 10:44:43.815506 14764 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:43.815696 14764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:43.816025 14764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:43.819319 14764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "a04680ee344f4c0aaa28616869ad9d0d"
format_stamp: "Formatted at 2026-09-28 10:44:43 on dist-test-slave-mn38"
server_key: "f22d16d60f9cfa30c9197e1a76a69d02"
server_key_iv: "854441648a8f7aa46dc4956aae89d866"
server_key_version: "encryptionkey@0"
I20260928 10:44:43.820192 14764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "a04680ee344f4c0aaa28616869ad9d0d"
format_stamp: "Formatted at 2026-09-28 10:44:43 on dist-test-slave-mn38"
server_key: "f22d16d60f9cfa30c9197e1a76a69d02"
server_key_iv: "854441648a8f7aa46dc4956aae89d866"
server_key_version: "encryptionkey@0"
I20260928 10:44:43.826356 14764 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:44:43.830451 14779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:43.832268 14764 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:44:43.832464 14764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "a04680ee344f4c0aaa28616869ad9d0d"
format_stamp: "Formatted at 2026-09-28 10:44:43 on dist-test-slave-mn38"
server_key: "f22d16d60f9cfa30c9197e1a76a69d02"
server_key_iv: "854441648a8f7aa46dc4956aae89d866"
server_key_version: "encryptionkey@0"
I20260928 10:44:43.832695 14764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:43.846738 14764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:43.851346 14764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:43.851645 14764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:43.852727 14764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:43.854213 14764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:43.854296 14764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:43.854417 14764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:43.854471 14764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:43.887611 14764 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:44157
I20260928 10:44:43.887657 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:44157 every 8 connection(s)
I20260928 10:44:43.889182 14764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:43.894199 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 14764
I20260928 10:44:43.894436 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:43.895270 9067 external_mini_cluster.cc:1428] Setting key d8073cfc25b6d01ae33354305c8cb728
Sep 28 10:44:43 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592283, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:43.912928 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:43.892410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:42117 (local address 127.8.218.254:36817)
0928 10:44:43.892900 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:43.892907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:43.896001 (+ 3094us) server_negotiation.cc:407] Connection header received
0928 10:44:43.896931 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:43.896937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:43.897026 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:43.897121 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:43.898983 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:43.899711 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:43.901348 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:43.901847 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:43.901987 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:43.905904 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:43.905910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:43.905914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:43.905949 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:43.909004 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:43.909973 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:43.909988 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:43.909992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:43.910148 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:43.910796 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:43.910802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:43.910805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:43.911220 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:43.911389 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:43.912293 (+ 904us) server_negotiation.cc:299] Negotiation successful
0928 10:44:43.912544 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":225,"threads_started":1}
I20260928 10:44:43.913744 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:43.893154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36817 (local address 127.8.218.193:42117)
0928 10:44:43.894197 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:43.894261 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:43.896600 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:43.897429 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:43.897476 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:43.898143 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:43.898790 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:43.898811 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:43.899868 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:43.899872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:43.901189 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:43.901203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:43.902123 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:43.902131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:43.902377 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:43.903092 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:43.903119 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:43.905644 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:43.909205 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:43.909216 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:43.909232 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:43.909824 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:43.910272 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:43.910276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:43.910278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:43.910637 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:43.911396 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:43.911405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:43.911814 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 10:44:43.912113 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 10:44:43.912545 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":396,"threads_started":1}
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:44:43.915477 14892 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36817
I20260928 10:44:43.915777 14892 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:43.916512 14892 heartbeater.cc:511] Master 127.8.218.254:36817 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
I20260928 10:44:43.918978 14703 ts_manager.cc:194] Registered new tserver with Master: a04680ee344f4c0aaa28616869ad9d0d (127.8.218.193:44157)
I20260928 10:44:43.920531 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:42117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:44:43.944944 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36817
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:44.287962 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:44.288285 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:44.288383 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:44.296108 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:44.296244 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:44.296397 14898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:44.306833 14898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:44.308122 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:44.309497 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:44.323884 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:44.323848 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:44.323912 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:44.324698 14898 server_base.cc:1034] running on GCE node
I20260928 10:44:44.325343 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:44.326442 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:44.327666 14898 hybrid_clock.cc:648] HybridClock initialized: now 1790592284327611 us; error 54 us; skew 500 ppm
I20260928 10:44:44.327982 14898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:44 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592284, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:44.334944 14898 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:44:44.336817 14898 webserver.cc:460] Webserver started at http://127.8.218.194:39835/ using document root <none> and password file <none>
I20260928 10:44:44.337664 14898 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:44.337806 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:44.338116 14898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:44.341187 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "18248f4c465e440fae929a51182277ba"
format_stamp: "Formatted at 2026-09-28 10:44:44 on dist-test-slave-mn38"
server_key: "7b9d315a3b270b53c592c35c7518b93d"
server_key_iv: "198fd7bbe5ff7185d651aa4081fdb704"
server_key_version: "encryptionkey@0"
I20260928 10:44:44.342075 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "18248f4c465e440fae929a51182277ba"
format_stamp: "Formatted at 2026-09-28 10:44:44 on dist-test-slave-mn38"
server_key: "7b9d315a3b270b53c592c35c7518b93d"
server_key_iv: "198fd7bbe5ff7185d651aa4081fdb704"
server_key_version: "encryptionkey@0"
I20260928 10:44:44.347332 14898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:44:44.350836 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:44.352203 14898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:44:44.352352 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "18248f4c465e440fae929a51182277ba"
format_stamp: "Formatted at 2026-09-28 10:44:44 on dist-test-slave-mn38"
server_key: "7b9d315a3b270b53c592c35c7518b93d"
server_key_iv: "198fd7bbe5ff7185d651aa4081fdb704"
server_key_version: "encryptionkey@0"
I20260928 10:44:44.352538 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:44.376134 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:44.380918 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:44.381222 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:44.382324 14898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:44.383898 14898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:44.383996 14898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:44.384121 14898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:44.384174 14898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:44.417397 14898 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:39997
I20260928 10:44:44.417747 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:39997 every 8 connection(s)
I20260928 10:44:44.419091 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:44.425844 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 14898
I20260928 10:44:44.426105 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:44.426539 9067 external_mini_cluster.cc:1428] Setting key 51b71b70110d2179efb8e9765f329317
Sep 28 10:44:44 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592284, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:44.440485 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:44.421933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:54297 (local address 127.8.218.254:36817)
0928 10:44:44.422365 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:44.422370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:44.423376 (+ 1006us) server_negotiation.cc:407] Connection header received
0928 10:44:44.425182 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:44.425189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:44.425278 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:44.425371 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:44.427120 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:44.427990 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:44.429843 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:44.430306 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:44.430426 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:44.434411 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:44.434416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:44.434420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:44.434455 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:44.437147 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:44.438108 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:44.438112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:44.438115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:44.438253 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:44.438706 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:44.438710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:44.438712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:44.438983 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:44.439114 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:44.439872 (+ 758us) server_negotiation.cc:299] Negotiation successful
0928 10:44:44.440105 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260928 10:44:44.441622 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:44.422376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36817 (local address 127.8.218.194:54297)
0928 10:44:44.423153 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:44.423242 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:44.424891 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:44.425667 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:44.425677 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:44.426285 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:44.426922 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:44.426948 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:44.428157 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:44.428170 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:44.429641 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:44.429655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:44.430557 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:44.430565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:44.430832 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:44.431557 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:44.431584 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:44.434104 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:44.437330 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:44.437341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:44.437359 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:44.437958 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:44.438369 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:44.438374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:44.438378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:44.438575 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:44.439110 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:44.439118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:44.439498 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 10:44:44.439749 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 10:44:44.440325 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":313,"threads_started":1}
I20260928 10:44:44.443423 15026 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36817
I20260928 10:44:44.443781 15026 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:44.444543 15026 heartbeater.cc:511] Master 127.8.218.254:36817 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:44:44.446157 14703 ts_manager.cc:194] Registered new tserver with Master: 18248f4c465e440fae929a51182277ba (127.8.218.194:39997)
I20260928 10:44:44.447077 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:54297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:44:44.475101 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36817
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:44.825013 15032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:44.825320 15032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:44.825416 15032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:44.832957 15032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:44.833092 15032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:44.833206 15032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:44.843743 15032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:44.845041 15032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:44.846347 15032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:44.860001 15037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:44.860061 15040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:44.860060 15038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:44.860375 15032 server_base.cc:1034] running on GCE node
I20260928 10:44:44.861204 15032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:44.862190 15032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:44.863400 15032 hybrid_clock.cc:648] HybridClock initialized: now 1790592284863333 us; error 55 us; skew 500 ppm
I20260928 10:44:44.863720 15032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:44 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592284, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:44.867065 15032 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:44:44.868798 15032 webserver.cc:460] Webserver started at http://127.8.218.195:34201/ using document root <none> and password file <none>
I20260928 10:44:44.869588 15032 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:44.869716 15032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:44.870023 15032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:44.873399 15032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "5205effced6b40a38011e1ee9ea4d982"
format_stamp: "Formatted at 2026-09-28 10:44:44 on dist-test-slave-mn38"
server_key: "a88367efd841222de55537fabd0831e7"
server_key_iv: "e2eb6be1131e9e78611846fe8ed7fd25"
server_key_version: "encryptionkey@0"
I20260928 10:44:44.874498 15032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "5205effced6b40a38011e1ee9ea4d982"
format_stamp: "Formatted at 2026-09-28 10:44:44 on dist-test-slave-mn38"
server_key: "a88367efd841222de55537fabd0831e7"
server_key_iv: "e2eb6be1131e9e78611846fe8ed7fd25"
server_key_version: "encryptionkey@0"
I20260928 10:44:44.880968 15032 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:44:44.885226 15047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:44.887035 15032 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:44.887259 15032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "5205effced6b40a38011e1ee9ea4d982"
format_stamp: "Formatted at 2026-09-28 10:44:44 on dist-test-slave-mn38"
server_key: "a88367efd841222de55537fabd0831e7"
server_key_iv: "e2eb6be1131e9e78611846fe8ed7fd25"
server_key_version: "encryptionkey@0"
I20260928 10:44:44.887591 15032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:44.904754 15032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:44.909619 15032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:44.909905 15032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:44.910933 15032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:44.912405 15032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:44.912496 15032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:44.912593 15032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:44.912637 15032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:44.924141 14892 heartbeater.cc:503] Master 127.8.218.254:36817 was elected leader, sending a full tablet report...
I20260928 10:44:44.944200 15032 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:33913
I20260928 10:44:44.944243 15159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:33913 every 8 connection(s)
I20260928 10:44:44.945514 15032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:44.946710 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 15032
I20260928 10:44:44.946890 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:44.947402 9067 external_mini_cluster.cc:1428] Setting key 82a94dc5f26b0807cf7f1dd097221bcd
Sep 28 10:44:44 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592284, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:44:44.965975 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:44.948385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:41023 (local address 127.8.218.254:36817)
0928 10:44:44.948821 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:44.948826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:44.949767 (+ 941us) server_negotiation.cc:407] Connection header received
0928 10:44:44.951500 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:44.951504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:44.951578 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:44.951662 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:44.953196 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:44.954093 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:44.955641 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:44.956108 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:44.956208 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:44.959610 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:44.959615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:44.959619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:44.959654 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:44.962425 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:44.963484 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:44.963489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:44.963491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:44.963620 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:44.964064 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:44.964068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:44.964071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:44.964350 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:44.964508 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:44.965286 (+ 778us) server_negotiation.cc:299] Negotiation successful
0928 10:44:44.965581 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":213,"threads_started":1}
I20260928 10:44:44.967218 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:44.948792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36817 (local address 127.8.218.195:41023)
0928 10:44:44.949609 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:44.949659 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:44.951268 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:44.951906 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:44.951916 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:44.952447 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:44.953013 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:44.953043 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:44.954224 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:44.954228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:44.955486 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:44.955499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:44.956294 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:44.956299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:44.956492 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:44.957149 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:44.957171 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:44.959384 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:44.962615 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:44.962626 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:44.962646 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:44.963355 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:44.963731 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:44.963736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:44.963740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:44.963954 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:44.964483 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:44.964492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:44.964899 (+ 407us) client_negotiation.cc:769] Sending connection context
0928 10:44:44.965166 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 10:44:44.965793 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":317,"threads_started":1}
I20260928 10:44:44.968688 15160 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36817
I20260928 10:44:44.969008 15160 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:44.969808 15160 heartbeater.cc:511] Master 127.8.218.254:36817 requested a full tablet report, sending...
I20260928 10:44:44.971422 14703 ts_manager.cc:194] Registered new tserver with Master: 5205effced6b40a38011e1ee9ea4d982 (127.8.218.195:33913)
I20260928 10:44:44.972285 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:41023
I20260928 10:44:44.975992 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/test-admin.keytab.
Sep 28 10:44:44 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:45.005981 9067 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 10:44:45.011292 15168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:45.008765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.8.218.193:44157)
0928 10:44:45.009489 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.009498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.009528 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:44:45.009598 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.009605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.009837 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.009986 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.010933 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43860: BlockingRecv error: recv got EOF from 127.0.0.1:43860 (error 108)
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":364,"threads_started":1}
W20260928 10:44:45.011926 9067 init.cc:244] Error while opening credential cache '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5cc)
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:45.013650 9067 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:45.015209 9067 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:44:45.032876 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:45.017246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.8.218.193:44157)
0928 10:44:45.017484 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.017489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.017615 (+ 126us) server_negotiation.cc:407] Connection header received
0928 10:44:45.017825 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.017830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.017911 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.018001 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.019063 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.020074 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.021412 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.022130 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.022244 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:45.026329 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:45.026346 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:45.026364 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:45.026420 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:45.029589 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.030417 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.030426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.030428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.030567 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.031045 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.031050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.031052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.031471 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:45.031717 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:45.032313 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 10:44:45.032601 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 10:44:45.039901 15168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:45.038107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.8.218.193:44157)
0928 10:44:45.038335 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.038342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.038453 (+ 111us) server_negotiation.cc:407] Connection header received
0928 10:44:45.038649 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.038655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.038732 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.038818 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.039649 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43880: BlockingRecv error: recv got EOF from 127.0.0.1:43880 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 10:44:45.040027 9067 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5cc' of type FILE
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:45.041661 9067 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:45.043120 9067 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:44:45.059454 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:45.044711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.8.218.193:44157)
0928 10:44:45.044959 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.044965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.045215 (+ 250us) server_negotiation.cc:407] Connection header received
0928 10:44:45.045360 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.045365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.045492 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.045593 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.046696 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.047457 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.048747 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.049188 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.049293 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:45.053208 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:45.053214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:45.053218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:45.053262 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:45.056081 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.056982 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.056988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.056990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.057218 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.057774 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.057778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.057780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.058204 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:45.058420 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:45.058866 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 10:44:45.059094 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260928 10:44:45.064479 15168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:45.062566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.8.218.193:44157)
0928 10:44:45.062790 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.062796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.062990 (+ 194us) server_negotiation.cc:407] Connection header received
0928 10:44:45.063158 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.063163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.063242 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.063324 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.064234 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43910: BlockingRecv error: recv got EOF from 127.0.0.1:43910 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 10:44:45.064610 9067 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790592230404718-9067-0/krb5kdc/krb5cc' of type FILE
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:45.066185 9067 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:45.067459 9067 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:44:45.082731 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:45.068787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.8.218.193:44157)
0928 10:44:45.069019 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.069025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.069302 (+ 277us) server_negotiation.cc:407] Connection header received
0928 10:44:45.069422 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.069426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.069536 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.069621 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.070624 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.071343 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.072681 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.073144 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.073247 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:45.077089 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:45.077095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:45.077099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:45.077143 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:45.079775 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.080630 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.080635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.080637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.080804 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.081221 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.081225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.081228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.081667 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:45.081933 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:45.082150 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 10:44:45.082342 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:44:45.084350 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 14764
I20260928 10:44:45.096138 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:45.085077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.8.218.193:44157)
0928 10:44:45.085214 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.085218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.085521 (+ 303us) server_negotiation.cc:407] Connection header received
0928 10:44:45.085674 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.085677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.085733 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.085800 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.086758 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.087463 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.088634 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.089077 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.089171 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:45.091222 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:45.091228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:45.091231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:45.091263 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:45.093878 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.094490 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.094494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.094497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.094654 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.094946 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.094950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.094953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.095223 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:45.095410 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:45.095682 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 10:44:45.095852 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 10:44:45.096709 14887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:45.183866 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 14764
I20260928 10:44:45.199760 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 14898
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:44:45.218415 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:45.200725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.8.218.194:39997)
0928 10:44:45.201265 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.201273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.201301 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:44:45.201364 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.201371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.201655 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.201820 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.202947 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.204028 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.205356 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.206203 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.206344 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:45.211301 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:45.211319 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:45.211337 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:45.211393 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:45.214632 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.215508 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.215516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.215519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.215681 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.216148 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.216153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.216155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.216816 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:45.217259 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:45.217760 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 10:44:45.218092 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":244,"threads_started":1}
I20260928 10:44:45.219870 15021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:45.303288 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 14898
I20260928 10:44:45.316329 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 15032
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[14658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:44:45.333426 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:45.317376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.8.218.195:33913)
0928 10:44:45.318046 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:45.318053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:45.318084 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:44:45.318144 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:45.318151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:45.318389 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:45.318540 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:45.319614 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.320665 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.322115 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:45.322728 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:45.322854 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:45.327140 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:45.327157 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:45.327178 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:45.327225 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:45.330119 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.331034 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.331050 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.331052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.331275 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:45.331798 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:45.331803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:45.331805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:45.332243 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:45.332499 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:45.332829 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 10:44:45.333057 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":358,"threads_started":1}
I20260928 10:44:45.335031 15155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:45.419559 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 15032
I20260928 10:44:45.432343 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 14671
I20260928 10:44:45.433560 14733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:45.536680 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 14671
2026-09-28T10:44:45Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3967 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[15188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[15188](info): set up 2 sockets
Sep 28 10:44:45 dist-test-slave-mn38 krb5kdc[15188](info): commencing operation
krb5kdc: starting...
W20260928 10:44:46.558928 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:44:46 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.254.keytab.
Entry for principal oryx/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.254.keytab.
I20260928 10:44:46.777231 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38155
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.254.keytab
--principal=oryx/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:47.119026 15201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:47.119343 15201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:47.119477 15201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:47.126909 15201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:47.127050 15201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:47.127128 15201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:47.127206 15201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:47.127267 15201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:47.137197 15201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.254.keytab
--principal=oryx/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:47.138674 15201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:47.140080 15201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:47.152474 15206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:47.152406 15207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:47.152452 15209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:47.153964 15201 server_base.cc:1034] running on GCE node
I20260928 10:44:47.154932 15201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:47.156641 15201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:47.157886 15201 hybrid_clock.cc:648] HybridClock initialized: now 1790592287157859 us; error 91 us; skew 500 ppm
I20260928 10:44:47.158313 15201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:47 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592287, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:47.161747 15201 init.cc:377] Logged in from keytab as oryx/127.8.218.254@KRBTEST.COM (short username oryx)
I20260928 10:44:47.163550 15201 webserver.cc:460] Webserver started at http://127.8.218.254:38621/ using document root <none> and password file <none>
I20260928 10:44:47.164345 15201 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:47.164482 15201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:47.164796 15201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:47.168154 15201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "4c02e72f8941457a9bf91b0c1f0a0081"
format_stamp: "Formatted at 2026-09-28 10:44:47 on dist-test-slave-mn38"
server_key: "f852af4dc6d3fcc5284b552b24351f54"
server_key_iv: "c1f7db44ad2400aaeaa90e11c03afe57"
server_key_version: "encryptionkey@0"
I20260928 10:44:47.169040 15201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "4c02e72f8941457a9bf91b0c1f0a0081"
format_stamp: "Formatted at 2026-09-28 10:44:47 on dist-test-slave-mn38"
server_key: "f852af4dc6d3fcc5284b552b24351f54"
server_key_iv: "c1f7db44ad2400aaeaa90e11c03afe57"
server_key_version: "encryptionkey@0"
I20260928 10:44:47.174793 15201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 10:44:47.178750 15216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:47.180761 15201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:44:47.180981 15201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "4c02e72f8941457a9bf91b0c1f0a0081"
format_stamp: "Formatted at 2026-09-28 10:44:47 on dist-test-slave-mn38"
server_key: "f852af4dc6d3fcc5284b552b24351f54"
server_key_iv: "c1f7db44ad2400aaeaa90e11c03afe57"
server_key_version: "encryptionkey@0"
I20260928 10:44:47.181245 15201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:47.204908 15201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:47.210620 15201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:47.210939 15201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:47.231524 15201 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:38155
I20260928 10:44:47.231536 15267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:38155 every 8 connection(s)
I20260928 10:44:47.233003 15201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:47.237020 15268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:47.240131 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 15201
I20260928 10:44:47.240371 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:47.241163 9067 external_mini_cluster.cc:1428] Setting key d2788567ecf9d6ef02617f010e1f357e
I20260928 10:44:47.250496 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: Bootstrap starting.
I20260928 10:44:47.253926 15268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:47.255002 15268 log.cc:824] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:44:47 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:47.258262 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: No bootstrap required, opened a new log
I20260928 10:44:47.264516 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:47.243591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.8.218.254:38155)
0928 10:44:47.244400 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:47.244414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:47.244459 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:44:47.246225 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:47.246252 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:47.246753 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:47.247174 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:47.248998 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:47.250574 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:47.251906 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:47.252659 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:47.252861 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:47.257047 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:47.257057 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:47.257074 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:47.257116 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:47.260254 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:47.260978 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:47.260987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:47.260995 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:47.261181 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:47.261645 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:47.261649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:47.261651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:47.262220 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:47.262526 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:47.262977 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 10:44:47.263435 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":364,"threads_started":1}
I20260928 10:44:47.266418 15268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } }
I20260928 10:44:47.266790 15268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:47.266889 15268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c02e72f8941457a9bf91b0c1f0a0081, State: Initialized, Role: FOLLOWER
I20260928 10:44:47.267503 15268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } }
I20260928 10:44:47.267720 15268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:47.267853 15268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:47.268005 15268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:47.270681 15268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } }
I20260928 10:44:47.271433 15268 leader_election.cc:302] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c02e72f8941457a9bf91b0c1f0a0081; no voters:
I20260928 10:44:47.272063 15268 leader_election.cc:288] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:47.272210 15273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:47.273289 15273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 1 LEADER]: Becoming Leader. State: Replica: 4c02e72f8941457a9bf91b0c1f0a0081, State: Running, Role: LEADER
I20260928 10:44:47.273866 15273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } }
I20260928 10:44:47.274353 15268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:47.277654 15275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c02e72f8941457a9bf91b0c1f0a0081. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } }
I20260928 10:44:47.278656 15275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:47.278131 15274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } }
I20260928 10:44:47.280019 15274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
W20260928 10:44:47.289276 15289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:44:47.289454 15289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:44:47.289588 15281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:47.291564 15281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:47.300786 15281 catalog_manager.cc:1347] Generated new cluster ID: 982ffa990fe5464ba4040ba7b39b98a7
I20260928 10:44:47.300942 15281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:47.327877 15281 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:47.329977 15281 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:47.345547 15281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: Generated new TSK 0
I20260928 10:44:47.346776 15281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab.
Entry for principal oryx/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab.
I20260928 10:44:47.405181 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38155
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab
--principal=oryx/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:47.767386 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:47.767712 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:47.767853 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:47.777208 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:47.777550 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:47.777842 15294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:47.790179 15294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab
--principal=oryx/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:47.791746 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:47.793241 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:47.808228 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:47.808883 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:47.808902 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:47.809525 15294 server_base.cc:1034] running on GCE node
I20260928 10:44:47.810489 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:47.811730 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:47.812997 15294 hybrid_clock.cc:648] HybridClock initialized: now 1790592287812928 us; error 64 us; skew 500 ppm
I20260928 10:44:47.813373 15294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:47 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592287, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:47.816872 15294 init.cc:377] Logged in from keytab as oryx/127.8.218.193@KRBTEST.COM (short username oryx)
I20260928 10:44:47.818758 15294 webserver.cc:460] Webserver started at http://127.8.218.193:38077/ using document root <none> and password file <none>
I20260928 10:44:47.819644 15294 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:47.819769 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:47.820055 15294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:47.823428 15294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "e3fd0e6225a848f88b069315eb44875a"
format_stamp: "Formatted at 2026-09-28 10:44:47 on dist-test-slave-mn38"
server_key: "a9c947b82eede17ef5c1a05c882cb4da"
server_key_iv: "4e82e9468d86b4a3c0a1737dd5f4cf50"
server_key_version: "encryptionkey@0"
I20260928 10:44:47.824290 15294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "e3fd0e6225a848f88b069315eb44875a"
format_stamp: "Formatted at 2026-09-28 10:44:47 on dist-test-slave-mn38"
server_key: "a9c947b82eede17ef5c1a05c882cb4da"
server_key_iv: "4e82e9468d86b4a3c0a1737dd5f4cf50"
server_key_version: "encryptionkey@0"
I20260928 10:44:47.831403 15294 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260928 10:44:47.835933 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:47.838244 15294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:44:47.838490 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "e3fd0e6225a848f88b069315eb44875a"
format_stamp: "Formatted at 2026-09-28 10:44:47 on dist-test-slave-mn38"
server_key: "a9c947b82eede17ef5c1a05c882cb4da"
server_key_iv: "4e82e9468d86b4a3c0a1737dd5f4cf50"
server_key_version: "encryptionkey@0"
I20260928 10:44:47.838754 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:47.882258 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:47.887573 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:47.887876 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:47.888917 15294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:47.890488 15294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:47.890568 15294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:47.890671 15294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:47.890726 15294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:47.923189 15294 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:33805
I20260928 10:44:47.923326 15421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:33805 every 8 connection(s)
I20260928 10:44:47.924618 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:47.931381 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 15294
I20260928 10:44:47.931594 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:47.932205 9067 external_mini_cluster.cc:1428] Setting key 83e36d9204c7cb54dfeb8a76a2069ef0
Sep 28 10:44:47 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592287, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:47.947494 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:47.927497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:36287 (local address 127.8.218.254:38155)
0928 10:44:47.928029 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:47.928036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:47.929033 (+ 997us) server_negotiation.cc:407] Connection header received
0928 10:44:47.930946 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:47.930954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:47.931051 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:47.931177 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:47.933016 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:47.933948 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:47.935987 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:47.936567 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:47.936703 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:47.940846 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:47.940855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:47.940860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:47.940908 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:47.943772 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:47.944944 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:47.944949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:47.944952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:47.945094 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:47.945670 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:47.945676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:47.945678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:47.946020 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:47.946144 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:47.947024 (+ 880us) server_negotiation.cc:299] Negotiation successful
0928 10:44:47.947223 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":264,"threads_started":1}
I20260928 10:44:47.949046 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:47.928090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.193:36287)
0928 10:44:47.928854 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:47.928901 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:47.930602 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:47.931424 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:47.931435 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:47.932088 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:47.932782 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:47.932823 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:47.934119 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:47.934124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:47.935786 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:47.935804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:47.936832 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:47.936839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:47.937070 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:47.937845 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:47.937871 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:47.940528 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:47.944013 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:47.944025 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:47.944049 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:47.944764 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:47.945216 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:47.945221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:47.945224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:47.945519 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:47.946153 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:47.946162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:47.946626 (+ 464us) client_negotiation.cc:769] Sending connection context
0928 10:44:47.946916 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 10:44:47.947503 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":288,"threads_started":1}
I20260928 10:44:47.950899 15422 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38155
I20260928 10:44:47.951247 15422 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:47.952073 15422 heartbeater.cc:511] Master 127.8.218.254:38155 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:44:47.954483 15233 ts_manager.cc:194] Registered new tserver with Master: e3fd0e6225a848f88b069315eb44875a (127.8.218.193:33805)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.194@KRBTEST.COM" created.
I20260928 10:44:47.956223 15233 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:36287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab.
Entry for principal oryx/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab.
I20260928 10:44:47.986303 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38155
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab
--principal=oryx/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:48.337625 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:48.337929 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:48.338068 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:48.345798 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:48.345933 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:48.346097 15428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:48.355947 15428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab
--principal=oryx/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:48.357303 15428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:48.358616 15428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:48.371628 15433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:48.371662 15434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:48.371683 15436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:48.372655 15428 server_base.cc:1034] running on GCE node
I20260928 10:44:48.373298 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:48.374460 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:48.375737 15428 hybrid_clock.cc:648] HybridClock initialized: now 1790592288375684 us; error 38 us; skew 500 ppm
I20260928 10:44:48.376121 15428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:48 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592288, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:48.379581 15428 init.cc:377] Logged in from keytab as oryx/127.8.218.194@KRBTEST.COM (short username oryx)
I20260928 10:44:48.381342 15428 webserver.cc:460] Webserver started at http://127.8.218.194:35283/ using document root <none> and password file <none>
I20260928 10:44:48.382179 15428 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:48.382324 15428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:48.382673 15428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:48.386159 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "56a6c2fd3ad94007b848eb02279fd43d"
format_stamp: "Formatted at 2026-09-28 10:44:48 on dist-test-slave-mn38"
server_key: "aed3f0fd92693737159036718d5ae628"
server_key_iv: "03e568ce55f840d59c780cf35170ad65"
server_key_version: "encryptionkey@0"
I20260928 10:44:48.387121 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "56a6c2fd3ad94007b848eb02279fd43d"
format_stamp: "Formatted at 2026-09-28 10:44:48 on dist-test-slave-mn38"
server_key: "aed3f0fd92693737159036718d5ae628"
server_key_iv: "03e568ce55f840d59c780cf35170ad65"
server_key_version: "encryptionkey@0"
I20260928 10:44:48.393421 15428 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260928 10:44:48.397603 15443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:48.399331 15428 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:44:48.399585 15428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "56a6c2fd3ad94007b848eb02279fd43d"
format_stamp: "Formatted at 2026-09-28 10:44:48 on dist-test-slave-mn38"
server_key: "aed3f0fd92693737159036718d5ae628"
server_key_iv: "03e568ce55f840d59c780cf35170ad65"
server_key_version: "encryptionkey@0"
I20260928 10:44:48.399812 15428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:48.418020 15428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:48.422688 15428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:48.422973 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:48.423965 15428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:48.425541 15428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:48.425619 15428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:48.425738 15428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:48.425786 15428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:48.455832 15428 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:41733
I20260928 10:44:48.455865 15555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:41733 every 8 connection(s)
I20260928 10:44:48.457115 15428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:48.458348 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 15428
I20260928 10:44:48.458523 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:48.458905 9067 external_mini_cluster.cc:1428] Setting key 84f9dad7b8431d1d3fba1c5ba770cc02
Sep 28 10:44:48 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592288, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.195@KRBTEST.COM" created.
I20260928 10:44:48.481094 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:48.460772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:40765 (local address 127.8.218.254:38155)
0928 10:44:48.461407 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:48.461413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:48.462460 (+ 1047us) server_negotiation.cc:407] Connection header received
0928 10:44:48.464410 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:48.464415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:48.464498 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:48.464616 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:48.466526 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:48.467410 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:48.469027 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:48.469473 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:48.469593 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:48.473595 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:48.473603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:48.473607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:48.473651 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:48.476785 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:48.477950 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:48.477956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:48.477960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:48.478140 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:48.478766 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:48.478772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:48.478775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:48.479139 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:48.479328 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:48.480455 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0928 10:44:48.480706 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":383,"threads_started":1}
I20260928 10:44:48.482229 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:48.461477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.194:40765)
0928 10:44:48.462283 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:48.462334 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:48.464095 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:48.464859 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:48.464871 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:48.465555 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:48.466295 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:48.466318 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:48.467579 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:48.467585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:48.468875 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:48.468889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:48.469701 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:48.469709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:48.469954 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:48.470614 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:48.470638 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:48.473263 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:48.477037 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:48.477048 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:48.477068 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:48.477714 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:48.478288 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:48.478293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:48.478295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:48.478564 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:48.479335 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:48.479346 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:48.479805 (+ 459us) client_negotiation.cc:769] Sending connection context
0928 10:44:48.480248 (+ 443us) client_negotiation.cc:240] Negotiation successful
0928 10:44:48.480801 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
I20260928 10:44:48.483912 15556 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38155
I20260928 10:44:48.484283 15556 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:48.485263 15556 heartbeater.cc:511] Master 127.8.218.254:38155 requested a full tablet report, sending...
I20260928 10:44:48.487092 15233 ts_manager.cc:194] Registered new tserver with Master: 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733)
I20260928 10:44:48.488229 15233 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:40765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab.
Entry for principal oryx/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab.
I20260928 10:44:48.508567 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38155
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab
--principal=oryx/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:48.849354 15562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:48.849691 15562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:48.849786 15562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:48.857045 15562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:48.857213 15562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:48.857348 15562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:48.867208 15562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab
--principal=oryx/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:48.868753 15562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:48.870152 15562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:48.884351 15570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:48.884325 15568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:48.884385 15567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:48.885535 15562 server_base.cc:1034] running on GCE node
I20260928 10:44:48.886350 15562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:48.887444 15562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:48.888675 15562 hybrid_clock.cc:648] HybridClock initialized: now 1790592288888615 us; error 77 us; skew 500 ppm
I20260928 10:44:48.889053 15562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:48 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592288, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:48.892925 15562 init.cc:377] Logged in from keytab as oryx/127.8.218.195@KRBTEST.COM (short username oryx)
I20260928 10:44:48.894650 15562 webserver.cc:460] Webserver started at http://127.8.218.195:36645/ using document root <none> and password file <none>
I20260928 10:44:48.895406 15562 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:48.895557 15562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:48.895877 15562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:48.899252 15562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "251b35975dd6493f99a3a2d66bdcf20b"
format_stamp: "Formatted at 2026-09-28 10:44:48 on dist-test-slave-mn38"
server_key: "2c629a3db368e2eaa2fe31d7438dcdb1"
server_key_iv: "94c411f649149885445bb6ff0a67d80e"
server_key_version: "encryptionkey@0"
I20260928 10:44:48.900170 15562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "251b35975dd6493f99a3a2d66bdcf20b"
format_stamp: "Formatted at 2026-09-28 10:44:48 on dist-test-slave-mn38"
server_key: "2c629a3db368e2eaa2fe31d7438dcdb1"
server_key_iv: "94c411f649149885445bb6ff0a67d80e"
server_key_version: "encryptionkey@0"
I20260928 10:44:48.906724 15562 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 10:44:48.911185 15577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:48.912814 15562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:44:48.913017 15562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "251b35975dd6493f99a3a2d66bdcf20b"
format_stamp: "Formatted at 2026-09-28 10:44:48 on dist-test-slave-mn38"
server_key: "2c629a3db368e2eaa2fe31d7438dcdb1"
server_key_iv: "94c411f649149885445bb6ff0a67d80e"
server_key_version: "encryptionkey@0"
I20260928 10:44:48.913259 15562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:48.941869 15562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:48.946399 15562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:48.946686 15562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:48.947702 15562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:48.949126 15562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:48.949205 15562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:48.949328 15562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:48.949380 15562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:48.960227 15422 heartbeater.cc:503] Master 127.8.218.254:38155 was elected leader, sending a full tablet report...
I20260928 10:44:48.980728 15562 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:33793
I20260928 10:44:48.980787 15689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:33793 every 8 connection(s)
I20260928 10:44:48.982219 15562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:48.989941 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 15562
I20260928 10:44:48.990161 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:48.990708 9067 external_mini_cluster.cc:1428] Setting key 0648b0179942c8c088d41bfd69a7e79b
Sep 28 10:44:48 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592288, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:49.004276 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:48.984853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:47899 (local address 127.8.218.254:38155)
0928 10:44:48.985318 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:48.985325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:48.986569 (+ 1244us) server_negotiation.cc:407] Connection header received
0928 10:44:48.988605 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:48.988611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:48.988696 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:48.988790 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:48.991030 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:48.992041 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:48.994109 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:48.994529 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:48.994629 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:48.998088 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:48.998094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:48.998097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:48.998133 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:49.000567 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.001404 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.001409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.001412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.001612 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.002359 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.002364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.002366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.002730 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:49.002902 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:49.003757 (+ 855us) server_negotiation.cc:299] Negotiation successful
0928 10:44:49.003974 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":206,"threads_started":1}
I20260928 10:44:49.005352 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:48.985581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.195:47899)
0928 10:44:48.986401 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:48.986443 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:48.988218 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:48.989060 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:48.989069 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:48.989806 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:48.990783 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:48.990813 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:48.992227 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:48.992232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:48.993971 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:48.993988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:48.994725 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:48.994732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:48.994965 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:48.995622 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:48.995644 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:48.997846 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:49.000725 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.000732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.000750 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.001233 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.001958 (+ 725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.001962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.001965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.002196 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.002883 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:49.002891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:49.003280 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 10:44:49.003537 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 10:44:49.004044 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":296,"threads_started":1}
I20260928 10:44:49.006881 15690 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38155
I20260928 10:44:49.007159 15690 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:49.007843 15690 heartbeater.cc:511] Master 127.8.218.254:38155 requested a full tablet report, sending...
I20260928 10:44:49.009305 15232 ts_manager.cc:194] Registered new tserver with Master: 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793)
I20260928 10:44:49.010215 15232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:47899
I20260928 10:44:49.019626 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.218.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.253.keytab.
Entry for principal oryx/127.8.218.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.253.keytab.
I20260928 10:44:49.068563 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.253:40141
--webserver_interface=127.8.218.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.253.keytab
--principal=oryx/127.8.218.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38155,127.8.218.253:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:49.422154 15696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:49.422458 15696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:49.422585 15696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:49.430218 15696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:49.430346 15696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:49.430406 15696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:49.430492 15696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:49.430549 15696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:49.440827 15696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38155,127.8.218.253:40141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.253.keytab
--principal=oryx/127.8.218.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.253:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.218.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
W20260928 10:44:49.441242 15696 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.218.254:38155,127.8.218.253:40141'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 10:44:49.442595 15696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:49.444155 15696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:49.456905 15701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:49.456959 15702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:49.457690 15704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:49.458148 15696 server_base.cc:1034] running on GCE node
I20260928 10:44:49.459010 15696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:49.460775 15696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:49.462081 15696 hybrid_clock.cc:648] HybridClock initialized: now 1790592289461967 us; error 100 us; skew 500 ppm
I20260928 10:44:49.462486 15696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:49 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592289, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:49.471717 15696 init.cc:377] Logged in from keytab as oryx/127.8.218.253@KRBTEST.COM (short username oryx)
I20260928 10:44:49.473274 15696 webserver.cc:460] Webserver started at http://127.8.218.253:37399/ using document root <none> and password file <none>
I20260928 10:44:49.474048 15696 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:49.474148 15696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:49.474502 15696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:49.478197 15696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data/instance:
uuid: "6be0af7dde2947cd98e2fae61320ea19"
format_stamp: "Formatted at 2026-09-28 10:44:49 on dist-test-slave-mn38"
server_key: "4311ea61ad2e14fa9ddf60f5bc9ad425"
server_key_iv: "abb30ce2a2411220c15ad888b6092070"
server_key_version: "encryptionkey@0"
I20260928 10:44:49.479269 15696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal/instance:
uuid: "6be0af7dde2947cd98e2fae61320ea19"
format_stamp: "Formatted at 2026-09-28 10:44:49 on dist-test-slave-mn38"
server_key: "4311ea61ad2e14fa9ddf60f5bc9ad425"
server_key_iv: "abb30ce2a2411220c15ad888b6092070"
server_key_version: "encryptionkey@0"
I20260928 10:44:49.485569 15696 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:44:49.489611 15711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:49.491123 15556 heartbeater.cc:503] Master 127.8.218.254:38155 was elected leader, sending a full tablet report...
I20260928 10:44:49.491206 15696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:44:49.491364 15696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
uuid: "6be0af7dde2947cd98e2fae61320ea19"
format_stamp: "Formatted at 2026-09-28 10:44:49 on dist-test-slave-mn38"
server_key: "4311ea61ad2e14fa9ddf60f5bc9ad425"
server_key_iv: "abb30ce2a2411220c15ad888b6092070"
server_key_version: "encryptionkey@0"
I20260928 10:44:49.491616 15696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:49.515080 15696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:49.519866 15696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:49.520139 15696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:49.537202 15696 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.253:40141
I20260928 10:44:49.537209 15762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.253:40141 every 8 connection(s)
I20260928 10:44:49.538612 15696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
I20260928 10:44:49.540313 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 15696
I20260928 10:44:49.543637 15763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:49.552296 15763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:44:49 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.253@KRBTEST.COM
Sep 28 10:44:49 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592289, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.253@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:49.565943 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.542825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.8.218.253:40141)
0928 10:44:49.543688 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:49.543704 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:49.543754 (+ 50us) server_negotiation.cc:407] Connection header received
0928 10:44:49.545485 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:49.545512 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:49.545916 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:49.546285 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:49.548131 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.549661 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.551490 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.552191 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.552336 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.557528 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:49.557538 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:49.557550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:49.557601 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:49.561239 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.562084 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.562093 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.562100 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.562307 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.562726 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.562731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.562733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.563576 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:49.563898 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:49.564322 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 10:44:49.564754 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":372,"threads_started":1}
I20260928 10:44:49.572337 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.555642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:45610)
0928 10:44:49.556389 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:49.556438 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:49.556598 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:49.557406 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:49.557413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:49.558221 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:49.558829 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.558850 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.560114 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.560118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.561375 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.561389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.562193 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.562200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.562379 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.563014 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:49.563035 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:49.565578 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:49.568748 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.568759 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.568766 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.569550 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.570155 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.570160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.570162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.570342 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.571262 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:49.571270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:49.571474 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 10:44:49.571667 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 10:44:49.571985 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":317,"threads_started":1}
I20260928 10:44:49.572413 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.555694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.8.218.254:38155)
0928 10:44:49.557091 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:49.557096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:49.557113 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:44:49.557162 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:49.557166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:49.557223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:49.557320 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:49.559052 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.559961 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.561566 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.561996 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.562115 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.565777 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:49.565783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:49.565787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:49.565826 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:49.568586 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.569735 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.569740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.569743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.569942 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.570599 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.570604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.570606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.571095 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:49.571241 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:49.571842 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 10:44:49.572066 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1293,"thread_start_us":1202,"threads_started":1}
I20260928 10:44:49.574533 15763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:44:49 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592289, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.253@KRBTEST.COM for oryx/127.8.218.253@KRBTEST.COM
I20260928 10:44:49.589006 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.575604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.0.0.1:36850)
0928 10:44:49.575847 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:49.575866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:49.576008 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:49.576388 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:49.576392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:49.576747 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:49.576934 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.576944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.578220 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.578224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.579137 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.579149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.579933 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.579938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.580083 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.580746 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:49.580767 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:49.583365 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:49.586716 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.586720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.586723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.587139 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.587534 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.587538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.587540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.587670 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.588166 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:49.588170 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:49.588314 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 10:44:49.588428 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 10:44:49.588662 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 10:44:49.589192 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.575546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.8.218.253:40141)
0928 10:44:49.575760 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:49.575766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:49.575956 (+ 190us) server_negotiation.cc:407] Connection header received
0928 10:44:49.576160 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:49.576166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:49.576230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:49.576319 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:49.577130 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.578053 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.579327 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.579768 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.579868 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.583617 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:49.583623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:49.583626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:49.583666 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:49.586534 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.587265 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.587269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.587271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.587426 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.587771 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.587775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.587777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.588059 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:49.588167 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:49.588533 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 10:44:49.588865 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":79}
I20260928 10:44:49.594456 15763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Bootstrap starting.
I20260928 10:44:49.598237 15763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:49.599586 15763 log.cc:824] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:49.603077 15763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: No bootstrap required, opened a new log
I20260928 10:44:49.610646 15763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } }
I20260928 10:44:49.611019 15763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:49.611128 15763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be0af7dde2947cd98e2fae61320ea19, State: Initialized, Role: FOLLOWER
I20260928 10:44:49.611784 15763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } }
I20260928 10:44:49.612885 15770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } } }
I20260928 10:44:49.613291 15770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:44:49.614051 15763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 10:44:49.624560 15781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:44:49.624742 15781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:44:49.627702 15696 master_runner.cc:428] Detected that this master 6be0af7dde2947cd98e2fae61320ea19 is joining an existing cluster
I20260928 10:44:49.627815 15696 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.218.253:40141
WARNING: no policy specified for oryx/127.8.218.252@KRBTEST.COM; defaulting to no policy
I20260928 10:44:49.650328 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.633383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:45624)
0928 10:44:49.634177 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:49.634200 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:49.634463 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:49.635105 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:49.635110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:49.635831 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:49.636291 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.636304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.637998 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.638005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.639910 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.639929 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.641243 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.641249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.641428 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.642216 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:49.642243 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:49.643412 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:49.647478 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.647483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.647487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.647963 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.648369 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.648373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.648377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.648534 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.649222 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:49.649227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:49.649430 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 10:44:49.649669 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 10:44:49.649942 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":474,"threads_started":1}
I20260928 10:44:49.651170 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.633378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.8.218.254:38155)
0928 10:44:49.633601 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:49.633608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:49.634318 (+ 710us) server_negotiation.cc:407] Connection header received
0928 10:44:49.634633 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:49.634639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:49.634757 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:49.634842 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:49.636532 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.637770 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.640122 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.640819 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.640919 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.643639 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:49.643646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:49.643651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:49.643695 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:49.647301 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.648111 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.648118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.648121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.648278 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.648695 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.648701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.648704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.649079 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:49.649287 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:49.650768 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0928 10:44:49.650939 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.252@KRBTEST.COM" created.
I20260928 10:44:49.657888 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.633479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.8.218.253:40141)
0928 10:44:49.633675 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:49.633681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:49.641362 (+ 7681us) server_negotiation.cc:407] Connection header received
0928 10:44:49.641862 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:49.641868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:49.641936 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:49.642019 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:49.643539 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.644591 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.646625 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.647297 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.647397 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.649755 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:49.649763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:49.649768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:49.649806 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:49.654358 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.655645 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.655650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.655654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.655819 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.656233 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.656238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.656241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.656603 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:49.656763 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:49.657243 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 10:44:49.657516 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:44:49.658924 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.634504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.0.0.1:36858)
0928 10:44:49.641264 (+ 6760us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:49.641286 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:49.641720 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:49.642216 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:49.642220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:49.642996 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:49.643375 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.643389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.644861 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.644866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.646451 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.646468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.647709 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.647715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.648028 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.648683 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:49.648705 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:49.649543 (+ 838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:49.654528 (+ 4985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.654533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.654537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.655514 (+ 977us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.655956 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.655960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.655964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.656129 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.656828 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:49.656832 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:49.656996 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 10:44:49.658134 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0928 10:44:49.658479 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6592,"thread_start_us":6853,"threads_started":1}
Sep 28 10:44:49 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592287, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.254@KRBTEST.COM for oryx/127.8.218.253@KRBTEST.COM
I20260928 10:44:49.675498 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.658637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.8.218.253:40141)
0928 10:44:49.659247 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:49.659253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:49.662119 (+ 2866us) server_negotiation.cc:407] Connection header received
0928 10:44:49.662407 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:49.662416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:49.662493 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:49.662573 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:49.663767 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.664518 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.665803 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.666278 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.666385 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.669653 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:49.669661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:49.669666 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:49.669708 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:49.672540 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.673317 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.673321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.673324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.673524 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.673916 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.673922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.673925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.674241 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:49.674375 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:49.674854 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 10:44:49.675103 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493}
I20260928 10:44:49.675840 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.658134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.0.0.1:36864)
0928 10:44:49.661979 (+ 3845us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:49.662021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:49.662193 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:49.662645 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:49.662653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:49.663300 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:49.663608 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.663619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.664643 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.664647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.665674 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.665687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.666432 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.666438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.666622 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.667224 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:49.667242 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:49.669356 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:49.672720 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.672739 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.672748 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.673176 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.673637 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.673643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.673647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.673800 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.674371 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:49.674381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:49.674567 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 10:44:49.674733 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 10:44:49.675169 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3618,"thread_start_us":3473,"threads_started":1}
I20260928 10:44:49.678068 15232 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.218.253:40141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19"
member_type: NON_VOTER
last_known_addr {
host: "127.8.218.253"
port: 40141
}
attrs {
promote: true
}
}
dest_uuid: "4c02e72f8941457a9bf91b0c1f0a0081"
cas_config_opid_index: -1
I20260928 10:44:49.679086 15232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } }
I20260928 10:44:49.682247 15794 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6be0af7dde2947cd98e2fae61320ea19 (127.8.218.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } }
I20260928 10:44:49.683579 15738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:49.685531 15794 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.218.253:40141
I20260928 10:44:49.686460 15696 master.cc:537] Master@127.8.218.253:40141 shutting down...
I20260928 10:44:49.686391 15793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } }
I20260928 10:44:49.686908 15793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:49.687911 15738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Refusing update from remote peer 4c02e72f8941457a9bf91b0c1f0a0081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 10:44:49.689570 15794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:44:49.695636 15220 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.252.keytab.
Entry for principal oryx/127.8.218.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.252.keytab.
W20260928 10:44:49.697755 15220 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 -> Peer 6be0af7dde2947cd98e2fae61320ea19 (127.8.218.253:40141): Couldn't send request to peer 6be0af7dde2947cd98e2fae61320ea19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 10:44:49.699456 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.252:45803
--webserver_interface=127.8.218.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.252.keytab
--principal=oryx/127.8.218.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38155,127.8.218.253:40141,127.8.218.252:45803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:44:49.702472 15696 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:49.703066 15696 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:49.703320 15696 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: stopping tablet replica
I20260928 10:44:49.711007 15696 master.cc:552] Master@127.8.218.253:40141 shutdown complete.
I20260928 10:44:49.711112 15696 master_runner.cc:305] Clearing existing system tablet
I20260928 10:44:49.711567 15696 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:49.711663 15696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:49.715013 15696 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 10:44:49.717119 15799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:49.717583 15696 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:49.718097 15696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 10:44:49.718214 15696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
uuid: "6be0af7dde2947cd98e2fae61320ea19"
format_stamp: "Formatted at 2026-09-28 10:44:49 on dist-test-slave-mn38"
server_key: "4311ea61ad2e14fa9ddf60f5bc9ad425"
server_key_iv: "abb30ce2a2411220c15ad888b6092070"
server_key_version: "encryptionkey@0"
I20260928 10:44:49.723690 15696 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:49.731248 15696 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 10:44:49.731472 15696 log.cc:1196] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 10:44:49.731974 15696 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Deleting consensus metadata
I20260928 10:44:49.732266 15696 master_runner.cc:315] Copying system tablet from 127.8.218.254:38155
I20260928 10:44:49.735023 15696 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: tablet copy: Beginning tablet copy session from remote peer at address 127.8.218.254:38155
I20260928 10:44:49.756641 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.735891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:45636)
0928 10:44:49.741945 (+ 6054us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:49.741983 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:49.742196 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:49.742622 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:49.742628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:49.743230 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:49.743525 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.743540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.745057 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.745064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.746629 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:49.746651 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.747679 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.747686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:49.747879 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.748653 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:49.748679 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:49.749830 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:49.753656 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.753663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.753667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.754171 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.754620 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:49.754624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:49.754628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:49.754792 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:49.755514 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:49.755519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:49.755717 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 10:44:49.755863 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 10:44:49.756187 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5882,"thread_start_us":5753,"threads_started":1}
I20260928 10:44:49.756646 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:49.735938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.8.218.254:38155)
0928 10:44:49.736134 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:49.736141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:49.742138 (+ 5997us) server_negotiation.cc:407] Connection header received
0928 10:44:49.742347 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:49.742353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:49.742445 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:49.742570 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:49.743721 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.744887 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.746821 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:49.747495 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:49.747639 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:49.750085 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:49.750092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:49.750097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:49.750139 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:49.753514 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.754318 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.754323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.754327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.754502 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:49.755035 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:49.755040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:49.755043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:49.755393 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:49.755541 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:49.756034 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 10:44:49.756273 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:44:49.758222 15253 tablet_copy_service.cc:140] P 4c02e72f8941457a9bf91b0c1f0a0081: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6be0af7dde2947cd98e2fae61320ea19 ({username='oryx', principal='oryx/127.8.218.253@KRBTEST.COM'} at 127.0.0.1:45636)
I20260928 10:44:49.758550 15253 tablet_copy_service.cc:161] P 4c02e72f8941457a9bf91b0c1f0a0081: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6be0af7dde2947cd98e2fae61320ea19 at {username='oryx', principal='oryx/127.8.218.253@KRBTEST.COM'} at 127.0.0.1:45636: session id = 6be0af7dde2947cd98e2fae61320ea19-00000000000000000000000000000000
I20260928 10:44:49.763053 15253 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 10:44:49.765385 15696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 10:44:49.772583 15696 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: tablet copy: Starting download of 0 data blocks...
I20260928 10:44:49.773016 15696 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: tablet copy: Starting download of 1 WAL segments...
I20260928 10:44:49.776404 15696 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 10:44:49.781522 15253 tablet_copy_service.cc:342] P 4c02e72f8941457a9bf91b0c1f0a0081: Request end of tablet copy session 6be0af7dde2947cd98e2fae61320ea19-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.218.253@KRBTEST.COM'} at 127.0.0.1:45636
I20260928 10:44:49.781792 15253 tablet_copy_service.cc:433] P 4c02e72f8941457a9bf91b0c1f0a0081: ending tablet copy session 6be0af7dde2947cd98e2fae61320ea19-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6be0af7dde2947cd98e2fae61320ea19
I20260928 10:44:49.797220 15696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:49.801113 15806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:49.801750 15807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:49.806137 15696 server_base.cc:1034] running on GCE node
W20260928 10:44:49.806754 15809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:49.807272 15696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:49.807991 15696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:49.809235 15696 hybrid_clock.cc:648] HybridClock initialized: now 1790592289809176 us; error 33 us; skew 500 ppm
I20260928 10:44:49.809533 15696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:49 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592289, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:49.812139 15696 init.cc:377] Logged in from keytab as oryx/127.8.218.253@KRBTEST.COM (short username oryx)
I20260928 10:44:49.817214 15696 webserver.cc:460] Webserver started at http://127.8.218.253:41491/ using document root <none> and password file <none>
I20260928 10:44:49.817658 15696 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:49.817754 15696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:49.821036 15696 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 10:44:49.823030 15816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:49.823490 15696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 10:44:49.823658 15696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
uuid: "6be0af7dde2947cd98e2fae61320ea19"
format_stamp: "Formatted at 2026-09-28 10:44:49 on dist-test-slave-mn38"
server_key: "4311ea61ad2e14fa9ddf60f5bc9ad425"
server_key_iv: "abb30ce2a2411220c15ad888b6092070"
server_key_version: "encryptionkey@0"
I20260928 10:44:49.823839 15696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:49.840608 15696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:49.845000 15696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:49.867333 15696 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.253:40141
I20260928 10:44:49.867373 15867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.253:40141 every 8 connection(s)
I20260928 10:44:49.868259 15696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
I20260928 10:44:49.874182 15868 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:44:49.878798 15868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Bootstrap starting.
I20260928 10:44:49.899400 15868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:44:49.899999 15868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Bootstrap complete.
I20260928 10:44:49.901141 15868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } }
I20260928 10:44:49.901422 15868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6be0af7dde2947cd98e2fae61320ea19, State: Initialized, Role: LEARNER
I20260928 10:44:49.901794 15868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } }
I20260928 10:44:49.902721 15871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } }
I20260928 10:44:49.903192 15871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:44:49.903191 15868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:49.914346 15882 catalog_manager.cc:1259] Loaded cluster ID: 982ffa990fe5464ba4040ba7b39b98a7
I20260928 10:44:49.914481 15882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: loading cluster ID for follower catalog manager: success
I20260928 10:44:49.918645 15882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: acquiring CA information for follower catalog manager: success
I20260928 10:44:49.921759 15882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:44:50.013031 15690 heartbeater.cc:503] Master 127.8.218.254:38155 was elected leader, sending a full tablet report...
W20260928 10:44:50.115510 15797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:50.115859 15797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:50.115995 15797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:50.123857 15797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:50.123996 15797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:50.124071 15797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:50.124145 15797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:50.124208 15797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:50.134778 15797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38155,127.8.218.253:40141,127.8.218.252:45803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.252.keytab
--principal=oryx/127.8.218.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.252:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.218.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:50.136228 15797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:50.137646 15797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:50.150252 15887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:50.150298 15888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:50.150439 15890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:50.151396 15797 server_base.cc:1034] running on GCE node
I20260928 10:44:50.152253 15797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:50.153991 15797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:50.155246 15797 hybrid_clock.cc:648] HybridClock initialized: now 1790592290155167 us; error 69 us; skew 500 ppm
I20260928 10:44:50.155622 15797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:50 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592290, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:50.158906 15797 init.cc:377] Logged in from keytab as oryx/127.8.218.252@KRBTEST.COM (short username oryx)
I20260928 10:44:50.160704 15797 webserver.cc:460] Webserver started at http://127.8.218.252:45535/ using document root <none> and password file <none>
I20260928 10:44:50.161507 15797 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:50.161638 15797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:50.161942 15797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:50.165176 15797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data/instance:
uuid: "74a949e40a7743c89969c2db112fbb27"
format_stamp: "Formatted at 2026-09-28 10:44:50 on dist-test-slave-mn38"
server_key: "c6b18c32a404784faefe5bbe004050b2"
server_key_iv: "a03076d758fc4651624cda1dc2f06cf5"
server_key_version: "encryptionkey@0"
I20260928 10:44:50.166085 15797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal/instance:
uuid: "74a949e40a7743c89969c2db112fbb27"
format_stamp: "Formatted at 2026-09-28 10:44:50 on dist-test-slave-mn38"
server_key: "c6b18c32a404784faefe5bbe004050b2"
server_key_iv: "a03076d758fc4651624cda1dc2f06cf5"
server_key_version: "encryptionkey@0"
I20260928 10:44:50.171795 15797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:44:50.176371 15897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:50.178050 15797 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:44:50.178270 15797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
uuid: "74a949e40a7743c89969c2db112fbb27"
format_stamp: "Formatted at 2026-09-28 10:44:50 on dist-test-slave-mn38"
server_key: "c6b18c32a404784faefe5bbe004050b2"
server_key_iv: "a03076d758fc4651624cda1dc2f06cf5"
server_key_version: "encryptionkey@0"
I20260928 10:44:50.178480 15797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:50.201927 15797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:50.207374 15797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:50.207667 15797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:50.228112 15797 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.252:45803
I20260928 10:44:50.228157 15948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.252:45803 every 8 connection(s)
I20260928 10:44:50.229763 15797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
I20260928 10:44:50.234428 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 15797
I20260928 10:44:50.234723 15949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:50.247318 15949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:44:50 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592290, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.252@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
Sep 28 10:44:50 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.252@KRBTEST.COM
I20260928 10:44:50.266842 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.251161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.8.218.254:38155)
0928 10:44:50.251351 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.251356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.252143 (+ 787us) server_negotiation.cc:407] Connection header received
0928 10:44:50.252690 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.252696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.252776 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.252894 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.254288 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.254997 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.256528 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.256907 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.257016 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.260888 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.260893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.260896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.260932 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.263606 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.264373 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.264377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.264380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.264528 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.264944 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.264948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.264950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.265230 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.265361 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.266291 (+ 930us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.266528 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:44:50.268092 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.251133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:45640)
0928 10:44:50.251966 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.252023 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.252390 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.253001 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.253007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.253714 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:50.254100 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.254121 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.255141 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.255145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.256388 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.256400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.257145 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.257153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.257346 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.258102 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:50.258126 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:50.260657 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:50.263748 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.263761 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.263779 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.264239 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.264640 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.264645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.264649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.264837 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.265361 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:50.265370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:50.265870 (+ 500us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.266187 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.266677 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":389,"threads_started":1}
I20260928 10:44:50.268443 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.236940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.8.218.252:45803)
0928 10:44:50.248890 (+ 11950us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.248911 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.249860 (+ 949us) server_negotiation.cc:407] Connection header received
0928 10:44:50.252299 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.252327 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.252647 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.252841 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.254409 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.255575 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.256836 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.257397 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.257700 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.262552 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.262568 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.262587 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.262680 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.265331 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.266177 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.266186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.266189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.266339 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.266744 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.266749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.266751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.267193 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.267421 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.267814 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.268038 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11486,"thread_start_us":11878,"threads_started":1}
I20260928 10:44:50.269765 15949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:44:50 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592290, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.252@KRBTEST.COM for oryx/127.8.218.253@KRBTEST.COM
I20260928 10:44:50.284920 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.270579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.8.218.253:40141)
0928 10:44:50.271158 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.271164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.271183 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:44:50.271241 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.271246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.271321 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.271431 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.272021 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.273008 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.274493 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.274976 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.275169 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.278868 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.278874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.278877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.278913 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.281774 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.282686 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.282711 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.282714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.282904 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.283295 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.283300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.283303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.283641 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.283811 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.284321 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.284541 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":294,"threads_started":1}
I20260928 10:44:50.284920 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.270434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.0.0.1:36866)
0928 10:44:50.270648 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.270667 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.270768 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.271453 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.271456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.271770 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:50.271903 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.271910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.273137 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.273141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.274242 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.274268 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.275215 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.275221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.275616 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.276377 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:50.276397 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:50.278610 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:50.281964 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.281969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.281972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.282540 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.283020 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.283026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.283028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.283180 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.283771 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:50.283776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:50.283964 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.284101 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.284537 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 10:44:50.286441 15949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:44:50 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592290, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.252@KRBTEST.COM for oryx/127.8.218.252@KRBTEST.COM
I20260928 10:44:50.300066 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.287371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.8.218.252:45803)
0928 10:44:50.287696 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.287701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.287720 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:44:50.287764 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.287767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.287824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.287911 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.288713 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.289425 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.290658 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.291065 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.291164 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.294688 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.294693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.294697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.294735 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.297370 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.298123 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.298128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.298130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.298298 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.298722 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.298726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.298728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.299025 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.299140 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.299517 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.299795 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":52}
I20260928 10:44:50.300068 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.287227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:45803 (local address 127.0.0.1:52734)
0928 10:44:50.287455 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.287472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.287565 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.287979 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.287982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.288358 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:50.288565 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.288574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.289607 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.289611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.290524 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.290536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.291220 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.291226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.291419 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.292162 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:50.292181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:50.294430 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:50.297571 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.297577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.297580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.297985 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.298430 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.298434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.298437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.298609 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.299154 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:50.299158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:50.299305 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.299419 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.299650 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 10:44:50.305274 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Bootstrap starting.
I20260928 10:44:50.308777 15949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:50.309938 15949 log.cc:824] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:50.312845 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: No bootstrap required, opened a new log
I20260928 10:44:50.322685 15949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } }
I20260928 10:44:50.323087 15949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:50.323208 15949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a949e40a7743c89969c2db112fbb27, State: Initialized, Role: FOLLOWER
I20260928 10:44:50.323843 15949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } }
I20260928 10:44:50.324958 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.317956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.0.0.1:36874)
0928 10:44:50.318471 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.318492 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.318657 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.319132 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.319138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.319177 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:50.319392 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.319403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.320568 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.320575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.322571 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.322590 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.323987 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.323996 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.324201 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.324277 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.324391 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.324575 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":235,"threads_started":1}
I20260928 10:44:50.324975 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.318235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.8.218.253:40141)
0928 10:44:50.318434 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.318439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.318598 (+ 159us) server_negotiation.cc:407] Connection header received
0928 10:44:50.318853 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.318858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.318957 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.319077 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:50.319571 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.320409 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.322742 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.323789 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.323914 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.324131 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.324500 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.324675 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:44:50.325013 15958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } } }
I20260928 10:44:50.325505 15958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:44:50.326146 15843 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 10:44:50.326471 15949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:50.326792 15871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c02e72f8941457a9bf91b0c1f0a0081. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } }
I20260928 10:44:50.327020 15871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: This master's current role is: LEARNER
W20260928 10:44:50.335920 15969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:44:50.336079 15969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:44:50.340011 15797 master_runner.cc:428] Detected that this master 74a949e40a7743c89969c2db112fbb27 is joining an existing cluster
I20260928 10:44:50.340150 15797 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.218.252:45803
I20260928 10:44:50.367110 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.344671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.8.218.254:38155)
0928 10:44:50.344927 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.344933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.350167 (+ 5234us) server_negotiation.cc:407] Connection header received
0928 10:44:50.350504 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.350511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.350600 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.350696 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.352687 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.353848 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.356306 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.356960 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.357048 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.360157 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.360164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.360169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.360210 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.363578 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.364508 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.364515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.364518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.364678 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.365171 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.365176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.365180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.365562 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.365676 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.366526 (+ 850us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.366745 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260928 10:44:50.367110 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.344178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.8.218.253:40141)
0928 10:44:50.344377 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.344383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.350608 (+ 6225us) server_negotiation.cc:407] Connection header received
0928 10:44:50.350859 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.350865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.350940 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.351012 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.352685 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.353855 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.356137 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.356765 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.356867 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.360157 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.360164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.360169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.360221 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.363538 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.364508 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.364514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.364518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.364678 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.365171 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.365177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.365180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.365559 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.365676 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.366486 (+ 810us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.366754 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:44:50.368353 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.345265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.0.0.1:36876)
0928 10:44:50.350495 (+ 5230us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.350513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.350707 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.351238 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.351242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.351865 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:50.352524 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.352534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.354030 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.354035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.355943 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.355958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.357225 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.357233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.357494 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.358344 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:50.358380 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:50.359902 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:50.363714 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.363719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.363723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.364374 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.364839 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.364844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.364848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.365046 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.365773 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:50.365778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:50.366158 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.367410 (+ 1252us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.367803 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5092,"mutex_wait_us":110,"spinlock_wait_cycles":43520,"thread_start_us":6658,"threads_started":1}
I20260928 10:44:50.368395 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.344198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:45646)
0928 10:44:50.349971 (+ 5773us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.350016 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.350330 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.350902 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.350906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.351738 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:50.352510 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.352524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.354360 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.354365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.356143 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.356159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.357322 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.357327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.357540 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.358423 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:50.358445 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:50.359902 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:50.363721 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.363726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.363729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.364366 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.364839 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.364844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.364848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.365048 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.365773 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:50.365778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:50.366158 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.367511 (+ 1353us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.367987 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5589,"mutex_wait_us":206,"spinlock_wait_cycles":77568,"thread_start_us":5399,"threads_started":1}
I20260928 10:44:50.369472 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.344542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.8.218.252:45803)
0928 10:44:50.344773 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.344779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.352389 (+ 7610us) server_negotiation.cc:407] Connection header received
0928 10:44:50.352631 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.352636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.352697 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.352755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.354068 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.355343 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.357410 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.358184 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.358289 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.361419 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.361426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.361430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.361504 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.365684 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.366558 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.366563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.366567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.366719 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.367108 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.367113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.367116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.367780 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.367946 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.368727 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.369095 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":98}
I20260928 10:44:50.370365 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.344213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:45803 (local address 127.0.0.1:52740)
0928 10:44:50.352289 (+ 8076us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.352309 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.352497 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.352916 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.352921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.353607 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:50.353918 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.353930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.355575 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.355581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.357245 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.357260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.358624 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.358632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.358805 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.359458 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:50.359513 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:50.361245 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:50.365839 (+ 4594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.365845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.365848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.366426 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.366855 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.366859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.366863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.367002 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.368042 (+ 1040us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:50.368047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:50.368217 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.369510 (+ 1293us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.369903 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7622,"mutex_wait_us":1200,"thread_start_us":4699,"threads_started":1}
I20260928 10:44:50.374430 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 15294
Sep 28 10:44:50 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592287, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.254@KRBTEST.COM for oryx/127.8.218.252@KRBTEST.COM
Sep 28 10:44:50 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.193@KRBTEST.COM
I20260928 10:44:50.391371 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.374166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:45803 (local address 127.0.0.1:52754)
0928 10:44:50.375015 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.375034 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.375149 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.376429 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.376435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.376980 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:50.377146 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.377156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.378283 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.378289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.379722 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.379741 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.380772 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.380778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.380968 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.381668 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:50.381686 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:50.384242 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:50.388363 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.388370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.388374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.388830 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.389382 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.389388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.389391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.389665 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.390381 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:50.390387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:50.390588 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.390723 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.391026 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733}
I20260928 10:44:50.392436 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.375807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.8.218.252:45803)
0928 10:44:50.376003 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.376010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.376031 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:44:50.376148 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.376154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.376266 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.376370 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.377315 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.378139 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.379932 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.380593 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.380716 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.384536 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.384545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.384550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.384596 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.388132 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.389035 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.389040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.389044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.389220 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.389841 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.389846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.389849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.390222 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.391700 (+ 1478us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.391843 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.392094 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:44:50.394037 15232 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.218.252:45803: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "74a949e40a7743c89969c2db112fbb27"
member_type: NON_VOTER
last_known_addr {
host: "127.8.218.252"
port: 45803
}
attrs {
promote: true
}
}
dest_uuid: "4c02e72f8941457a9bf91b0c1f0a0081"
cas_config_opid_index: 5
I20260928 10:44:50.395123 15232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } }
I20260928 10:44:50.397068 15956 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 74a949e40a7743c89969c2db112fbb27 (127.8.218.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.398056 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.375487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.8.218.193:33805)
0928 10:44:50.376139 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.376148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.376179 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:44:50.376239 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.376247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.376522 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.376701 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.378399 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.379973 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.382103 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.383201 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.383350 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.389062 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.389084 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.389109 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.389165 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.393473 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.394664 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.394674 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.394677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.394856 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.395358 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.395363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.395367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.395839 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.396172 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.397207 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.397674 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":373,"threads_started":1}
I20260928 10:44:50.398332 15924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:50.400178 15417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:50.402271 15843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 LEARNER]: Refusing update from remote peer 4c02e72f8941457a9bf91b0c1f0a0081: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 10:44:50.402316 15924 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 FOLLOWER]: Refusing update from remote peer 4c02e72f8941457a9bf91b0c1f0a0081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 10:44:50.411468 15981 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 10:44:50.417615 15979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:50.419044 15956 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.218.252:45803
I20260928 10:44:50.419835 15797 master.cc:537] Master@127.8.218.252:45803 shutting down...
I20260928 10:44:50.419682 15982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.421273 15982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:50.426133 15843 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 74a949e40a7743c89969c2db112fbb27 (127.8.218.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.427475 15924 raft_consensus.cc:858] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } }
W20260928 10:44:50.427980 15924 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } }
I20260928 10:44:50.430374 15871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.430666 15871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:44:50.431885 15979 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: attempting to promote NON_VOTER 6be0af7dde2947cd98e2fae61320ea19 to VOTER
I20260928 10:44:50.432688 15979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } }
I20260928 10:44:50.437827 15843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 LEARNER]: Refusing update from remote peer 4c02e72f8941457a9bf91b0c1f0a0081: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 10:44:50.438584 15982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 10:44:50.442787 15979 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6be0af7dde2947cd98e2fae61320ea19 (127.8.218.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.447335 15979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.447757 15979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:50.448374 15981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.448814 15981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:50.450085 15842 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6be0af7dde2947cd98e2fae61320ea19 (127.8.218.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
W20260928 10:44:50.458149 15217 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 -> Peer 74a949e40a7743c89969c2db112fbb27 (127.8.218.252:45803): Couldn't send request to peer 74a949e40a7743c89969c2db112fbb27. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 10:44:50.460734 15871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.461031 15871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:44:50.470305 15924 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6be0af7dde2947cd98e2fae61320ea19 (127.8.218.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } }
I20260928 10:44:50.478981 15989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } }
I20260928 10:44:50.479305 15989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 10:44:50.479645 15958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c02e72f8941457a9bf91b0c1f0a0081. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } }
I20260928 10:44:50.479866 15958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 10:44:50.481565 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: NON_VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: true } } }
I20260928 10:44:50.482569 15797 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 10:44:50.482636 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 10:44:50.483135 15797 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 10:44:50.483348 15797 tablet_replica.cc:330] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: stopping tablet replica
I20260928 10:44:50.492723 15797 master.cc:552] Master@127.8.218.252:45803 shutdown complete.
I20260928 10:44:50.492841 15797 master_runner.cc:305] Clearing existing system tablet
I20260928 10:44:50.493253 15797 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:50.493382 15797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:50.496477 15797 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 10:44:50.498546 15994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:50.499086 15797 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:50.499366 15797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 10:44:50.499473 15797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
uuid: "74a949e40a7743c89969c2db112fbb27"
format_stamp: "Formatted at 2026-09-28 10:44:50 on dist-test-slave-mn38"
server_key: "c6b18c32a404784faefe5bbe004050b2"
server_key_iv: "a03076d758fc4651624cda1dc2f06cf5"
server_key_version: "encryptionkey@0"
I20260928 10:44:50.504966 15797 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:50.513566 15797 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 10:44:50.513788 15797 log.cc:1196] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 10:44:50.514359 15797 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Deleting consensus metadata
I20260928 10:44:50.514647 15797 master_runner.cc:315] Copying system tablet from 127.8.218.254:38155
I20260928 10:44:50.517882 15797 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: tablet copy: Beginning tablet copy session from remote peer at address 127.8.218.254:38155
I20260928 10:44:50.535847 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.519330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.8.218.254:38155)
0928 10:44:50.519565 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.519572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.519805 (+ 233us) server_negotiation.cc:407] Connection header received
0928 10:44:50.519995 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.520002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.520110 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.520242 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:50.521388 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.522649 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.524841 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.525796 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.525993 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.528750 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:50.528758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:50.528764 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:50.528811 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:50.532582 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.533612 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.533619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.533623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.533790 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:50.534290 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:50.534295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:50.534298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:50.534673 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:50.534825 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.535288 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.535517 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 10:44:50.536247 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.518956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:45656)
0928 10:44:50.519665 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.519690 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.519867 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.520294 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.520299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.520890 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:50.521171 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.521186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.522868 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.522875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.524580 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.524617 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.526048 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.526057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.526303 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.527246 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:50.527274 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:50.528480 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:50.532805 (+ 4325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.532812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.532817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.533391 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.533943 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:50.533949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:50.533953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:50.534164 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:50.534811 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:50.534816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:50.535019 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.535180 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.535537 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":466,"threads_started":1}
I20260928 10:44:50.537001 15253 tablet_copy_service.cc:140] P 4c02e72f8941457a9bf91b0c1f0a0081: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 74a949e40a7743c89969c2db112fbb27 ({username='oryx', principal='oryx/127.8.218.252@KRBTEST.COM'} at 127.0.0.1:45656)
I20260928 10:44:50.537184 15253 tablet_copy_service.cc:161] P 4c02e72f8941457a9bf91b0c1f0a0081: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 74a949e40a7743c89969c2db112fbb27 at {username='oryx', principal='oryx/127.8.218.252@KRBTEST.COM'} at 127.0.0.1:45656: session id = 74a949e40a7743c89969c2db112fbb27-00000000000000000000000000000000
I20260928 10:44:50.540827 15253 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 10:44:50.542840 15797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 10:44:50.549858 15797 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: tablet copy: Starting download of 0 data blocks...
I20260928 10:44:50.550171 15797 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: tablet copy: Starting download of 1 WAL segments...
I20260928 10:44:50.550573 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 15294
I20260928 10:44:50.554481 15797 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 10:44:50.558148 15253 tablet_copy_service.cc:342] P 4c02e72f8941457a9bf91b0c1f0a0081: Request end of tablet copy session 74a949e40a7743c89969c2db112fbb27-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.218.252@KRBTEST.COM'} at 127.0.0.1:45656
I20260928 10:44:50.558300 15253 tablet_copy_service.cc:433] P 4c02e72f8941457a9bf91b0c1f0a0081: ending tablet copy session 74a949e40a7743c89969c2db112fbb27-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 74a949e40a7743c89969c2db112fbb27
I20260928 10:44:50.569414 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:33805
--local_ip_for_outbound_sockets=127.8.218.193
--tserver_master_addrs=127.8.218.254:38155
--webserver_port=38077
--webserver_interface=127.8.218.193
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab
--principal=oryx/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141,127.8.218.252:45803 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:44:50.571473 15797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:50.574785 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:50.577099 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:50.585605 15797 server_base.cc:1034] running on GCE node
W20260928 10:44:50.586396 16005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:50.586865 15797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:50.587536 15797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:50.588747 15797 hybrid_clock.cc:648] HybridClock initialized: now 1790592290588697 us; error 36 us; skew 500 ppm
I20260928 10:44:50.589006 15797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:50 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592290, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:50.591365 15797 init.cc:377] Logged in from keytab as oryx/127.8.218.252@KRBTEST.COM (short username oryx)
I20260928 10:44:50.592793 15797 webserver.cc:460] Webserver started at http://127.8.218.252:34187/ using document root <none> and password file <none>
I20260928 10:44:50.593137 15797 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:50.593240 15797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:50.596683 15797 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 10:44:50.599327 16012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:50.599871 15797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 10:44:50.600049 15797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
uuid: "74a949e40a7743c89969c2db112fbb27"
format_stamp: "Formatted at 2026-09-28 10:44:50 on dist-test-slave-mn38"
server_key: "c6b18c32a404784faefe5bbe004050b2"
server_key_iv: "a03076d758fc4651624cda1dc2f06cf5"
server_key_version: "encryptionkey@0"
I20260928 10:44:50.600481 15797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:50.638491 15797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:50.643308 15797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:50.667397 15797 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.252:45803
I20260928 10:44:50.667577 16063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.252:45803 every 8 connection(s)
I20260928 10:44:50.668435 15797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
I20260928 10:44:50.673681 16064 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:44:50.677664 16064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Bootstrap starting.
I20260928 10:44:50.695446 16064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:44:50.696015 16064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: Bootstrap complete.
I20260928 10:44:50.697278 16064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } }
I20260928 10:44:50.697641 16064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74a949e40a7743c89969c2db112fbb27, State: Initialized, Role: LEARNER
I20260928 10:44:50.697899 16064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } }
I20260928 10:44:50.698801 16067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.699178 16064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:50.699342 16067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: LEARNER
I20260928 10:44:50.709143 16078 catalog_manager.cc:1259] Loaded cluster ID: 982ffa990fe5464ba4040ba7b39b98a7
I20260928 10:44:50.709285 16078 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: loading cluster ID for follower catalog manager: success
I20260928 10:44:50.712285 16078 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: acquiring CA information for follower catalog manager: success
I20260928 10:44:50.714643 16078 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:44:50.831295 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.822432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:45803 (local address 127.0.0.1:52766)
0928 10:44:50.822707 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:50.822729 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:50.822921 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:50.823991 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:50.823998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:50.824036 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:50.824266 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.824279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.826487 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.826495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.828392 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:50.828415 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.830410 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.830417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:50.830638 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.830680 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 10:44:50.830799 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 10:44:50.830964 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 10:44:50.831833 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:50.822641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.8.218.252:45803)
0928 10:44:50.823562 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:50.823568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:50.823596 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:44:50.823666 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:50.823671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:50.823768 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:50.823915 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:50.824620 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.826267 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.828639 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:50.830160 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:50.830338 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:50.830689 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:50.830837 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 10:44:50.831005 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":588,"threads_started":1}
I20260928 10:44:50.832901 16039 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 10:44:50.833582 16067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c02e72f8941457a9bf91b0c1f0a0081. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:50.833907 16067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: LEARNER
W20260928 10:44:50.992388 16003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:50.992715 16003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:50.992854 16003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:51.000674 16003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:51.000813 16003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:51.000972 16003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:51.011569 16003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab
--principal=oryx/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:33805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=38077
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141,127.8.218.252:45803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:51.013026 16003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:51.014593 16003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:51.030731 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:51.030797 16085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:51.030817 16087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:51.031680 16003 server_base.cc:1034] running on GCE node
I20260928 10:44:51.032330 16003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:51.033380 16003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:51.034691 16003 hybrid_clock.cc:648] HybridClock initialized: now 1790592291034620 us; error 56 us; skew 500 ppm
I20260928 10:44:51.035059 16003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592291, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:51.039381 16003 init.cc:377] Logged in from keytab as oryx/127.8.218.193@KRBTEST.COM (short username oryx)
I20260928 10:44:51.041116 16003 webserver.cc:460] Webserver started at http://127.8.218.193:38077/ using document root <none> and password file <none>
I20260928 10:44:51.041939 16003 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:51.042075 16003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:51.048213 16003 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:44:51.052027 16094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:51.053996 16003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:51.054220 16003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "e3fd0e6225a848f88b069315eb44875a"
format_stamp: "Formatted at 2026-09-28 10:44:47 on dist-test-slave-mn38"
server_key: "a9c947b82eede17ef5c1a05c882cb4da"
server_key_iv: "4e82e9468d86b4a3c0a1737dd5f4cf50"
server_key_version: "encryptionkey@0"
I20260928 10:44:51.055414 16003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:51.079866 16003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:51.085083 16003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:51.085382 16003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:51.086620 16003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:51.088336 16003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:51.088457 16003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:51.088605 16003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:51.088696 16003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:51.124431 16003 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:33805
I20260928 10:44:51.124533 16206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:33805 every 8 connection(s)
I20260928 10:44:51.125998 16003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:51.131465 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 16003
I20260928 10:44:51.131794 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 15428
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592291, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.252@KRBTEST.COM
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592291, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.253@KRBTEST.COM
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.194@KRBTEST.COM
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592291, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:51.169816 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.129792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:34865 (local address 127.8.218.252:45803)
0928 10:44:51.130181 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:51.130188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:51.146310 (+ 16122us) server_negotiation.cc:407] Connection header received
0928 10:44:51.147506 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:51.147513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:51.147606 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:51.147730 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:51.150654 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.151858 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.154676 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.155401 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.155519 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.161923 (+ 6404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:51.161932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:51.161937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:51.161986 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:51.165658 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.166951 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.166957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.166960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.167118 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.167736 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.167741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.167743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.168083 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:51.168217 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:51.169179 (+ 962us) server_negotiation.cc:299] Negotiation successful
0928 10:44:51.169480 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260928 10:44:51.169836 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.129422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:40105 (local address 127.8.218.253:40141)
0928 10:44:51.140989 (+ 11567us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:51.140996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:51.145509 (+ 4513us) server_negotiation.cc:407] Connection header received
0928 10:44:51.148349 (+ 2840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:51.148354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:51.148440 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:51.148534 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:51.150629 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.151757 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.154641 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.155297 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.155397 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.161936 (+ 6539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:51.161945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:51.161950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:51.161994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:51.165850 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.166944 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.166951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.166955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.167107 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.167728 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.167733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.167736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.168043 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:51.168145 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:51.169158 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0928 10:44:51.169424 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11410,"thread_start_us":11261,"threads_started":1}
I20260928 10:44:51.172148 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.133383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.8.218.194:41733)
0928 10:44:51.149964 (+ 16581us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:51.149973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:51.150723 (+ 750us) server_negotiation.cc:407] Connection header received
0928 10:44:51.150800 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:51.150809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:51.151096 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:51.151244 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:51.153430 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.154968 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.156834 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.157715 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.157850 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.163867 (+ 6017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:51.163889 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:51.163910 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:51.163955 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:51.168522 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.169539 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.169549 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.169553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.169717 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.170281 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.170286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.170289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.170675 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:51.170975 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:51.171287 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 10:44:51.171554 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16384,"thread_start_us":16194,"threads_started":1}
I20260928 10:44:51.172263 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.130094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.8.218.193:40105)
0928 10:44:51.141646 (+ 11552us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:51.141724 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:51.145183 (+ 3459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:51.148587 (+ 3404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:51.148591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:51.149233 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:51.150392 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.150424 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.151993 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.151998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.154462 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.154479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.155838 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.155846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.156327 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.157327 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:51.157370 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:51.161612 (+ 4242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:51.166010 (+ 4398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.166015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.166019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.166791 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.167265 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.167269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.167273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.167585 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.168222 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:51.168228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:51.168722 (+ 494us) client_negotiation.cc:769] Sending connection context
0928 10:44:51.170103 (+ 1381us) client_negotiation.cc:240] Negotiation successful
0928 10:44:51.170724 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11052,"mutex_wait_us":356,"spinlock_wait_cycles":33536,"thread_start_us":15489,"threads_started":1}
I20260928 10:44:51.172933 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.130094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:45803 (local address 127.8.218.193:34865)
0928 10:44:51.146141 (+ 16047us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:51.146165 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:51.147252 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:51.148236 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:51.148252 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:51.149238 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:51.150394 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.150424 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.152052 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.152058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.154461 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.154479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.155839 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.155845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.156329 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.157468 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:51.157512 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:51.161614 (+ 4102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:51.165849 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.165860 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.165883 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.166800 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.167272 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.167277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.167281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.167598 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.168298 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:51.168303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:51.168701 (+ 398us) client_negotiation.cc:769] Sending connection context
0928 10:44:51.170048 (+ 1347us) client_negotiation.cc:240] Negotiation successful
0928 10:44:51.171019 (+ 971us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15832,"mutex_wait_us":680,"spinlock_wait_cycles":251776,"thread_start_us":17418,"threads_started":1}
I20260928 10:44:51.173913 15551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:51.174445 16207 heartbeater.cc:347] Connected to a master server at 127.8.218.252:45803
I20260928 10:44:51.174469 16209 heartbeater.cc:347] Connected to a master server at 127.8.218.253:40141
I20260928 10:44:51.174878 16207 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:51.174885 16209 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:51.176034 16209 heartbeater.cc:511] Master 127.8.218.253:40141 requested a full tablet report, sending...
I20260928 10:44:51.176061 16207 heartbeater.cc:511] Master 127.8.218.252:45803 requested a full tablet report, sending...
I20260928 10:44:51.179112 15833 ts_manager.cc:194] Registered new tserver with Master: e3fd0e6225a848f88b069315eb44875a (127.8.218.193:33805)
I20260928 10:44:51.179281 16029 ts_manager.cc:194] Registered new tserver with Master: e3fd0e6225a848f88b069315eb44875a (127.8.218.193:33805)
I20260928 10:44:51.181856 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.130291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.193:34177)
0928 10:44:51.148039 (+ 17748us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:51.148056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:51.148150 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:51.158783 (+ 10633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:51.158788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:51.159359 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:51.159671 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.159686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.162724 (+ 3038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.162729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.164199 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.164217 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.165321 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.165329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.165545 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.166076 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:51.166102 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:51.169109 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:51.178166 (+ 9057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.178172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.178177 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.178767 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.179356 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.179362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.179366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.179547 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.180723 (+ 1176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:51.180729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:51.180978 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 10:44:51.181157 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 10:44:51.181533 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17619,"mutex_wait_us":20,"spinlock_wait_cycles":64768,"thread_start_us":16033,"threads_started":1}
I20260928 10:44:51.182734 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.129996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:34177 (local address 127.8.218.254:38155)
0928 10:44:51.158405 (+ 28409us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:51.158411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:51.158431 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:44:51.158494 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:51.158499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:51.158581 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:51.158959 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:51.159946 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.162543 (+ 2597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.164426 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.165091 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.165833 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.169344 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:51.169351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:51.169356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:51.169399 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:51.177960 (+ 8561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.179012 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.179018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.179022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.179193 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.179707 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.179711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.179715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.180132 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:51.180724 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:51.182213 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0928 10:44:51.182422 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28264,"thread_start_us":28144,"threads_started":1}
I20260928 10:44:51.183607 16208 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38155
I20260928 10:44:51.183802 16208 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:51.184303 16208 heartbeater.cc:511] Master 127.8.218.254:38155 requested a full tablet report, sending...
I20260928 10:44:51.185781 15232 ts_manager.cc:194] Re-registered known tserver with Master: e3fd0e6225a848f88b069315eb44875a (127.8.218.193:33805)
I20260928 10:44:51.186790 15232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:34177
I20260928 10:44:51.267577 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 15428
I20260928 10:44:51.280910 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:41733
--local_ip_for_outbound_sockets=127.8.218.194
--tserver_master_addrs=127.8.218.254:38155
--webserver_port=35283
--webserver_interface=127.8.218.194
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab
--principal=oryx/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141,127.8.218.252:45803 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:44:51.330314 15982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: NON_VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: true } } }
I20260928 10:44:51.330688 15982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:51.332261 15982 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: attempting to promote NON_VOTER 74a949e40a7743c89969c2db112fbb27 to VOTER
I20260928 10:44:51.333076 15982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } }
I20260928 10:44:51.335284 16039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 LEARNER]: Refusing update from remote peer 4c02e72f8941457a9bf91b0c1f0a0081: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 10:44:51.335412 15842 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Refusing update from remote peer 4c02e72f8941457a9bf91b0c1f0a0081: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 10:44:51.335884 15982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 10:44:51.336294 16220 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 10:44:51.339335 16221 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 74a949e40a7743c89969c2db112fbb27 (127.8.218.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:51.339927 15842 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 74a949e40a7743c89969c2db112fbb27 (127.8.218.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:51.339998 16039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 74a949e40a7743c89969c2db112fbb27 (127.8.218.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:51.343318 16220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:51.343606 16220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:51.343994 16220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:51.344242 16220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:51.344552 16220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:51.344758 16220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:51.345911 16225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:51.346274 16225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:44:51.346823 16224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:51.347291 16224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 10:44:51.632360 16219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:51.632681 16219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:51.632797 16219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:51.640472 16219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:51.640618 16219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:51.640770 16219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:51.651026 16219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab
--principal=oryx/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:41733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=35283
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141,127.8.218.252:45803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:51.652438 16219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:51.653934 16219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:51.668113 16233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:51.668104 16230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:51.668860 16219 server_base.cc:1034] running on GCE node
W20260928 10:44:51.670361 16231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:51.671192 16219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:51.672425 16219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:51.673641 16219 hybrid_clock.cc:648] HybridClock initialized: now 1790592291673576 us; error 60 us; skew 500 ppm
I20260928 10:44:51.674060 16219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592291, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:51.678406 16219 init.cc:377] Logged in from keytab as oryx/127.8.218.194@KRBTEST.COM (short username oryx)
I20260928 10:44:51.680121 16219 webserver.cc:460] Webserver started at http://127.8.218.194:35283/ using document root <none> and password file <none>
I20260928 10:44:51.680922 16219 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:51.681100 16219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:51.687580 16219 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:44:51.690912 16240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:51.692662 16219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:44:51.692862 16219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "56a6c2fd3ad94007b848eb02279fd43d"
format_stamp: "Formatted at 2026-09-28 10:44:48 on dist-test-slave-mn38"
server_key: "aed3f0fd92693737159036718d5ae628"
server_key_iv: "03e568ce55f840d59c780cf35170ad65"
server_key_version: "encryptionkey@0"
I20260928 10:44:51.693998 16219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:51.724709 16219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:51.729799 16219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:51.730059 16219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:51.731174 16219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:51.732769 16219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:51.732861 16219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:51.732961 16219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:51.733031 16219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:51.762684 16219 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:41733
I20260928 10:44:51.762738 16352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:41733 every 8 connection(s)
I20260928 10:44:51.764112 16219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:51.773536 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 16219
I20260928 10:44:51.773845 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 15562
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592291, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592291, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.253@KRBTEST.COM
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592291, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.252@KRBTEST.COM
I20260928 10:44:51.794082 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.767465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:35861 (local address 127.8.218.253:40141)
0928 10:44:51.768886 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:51.768893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:51.770980 (+ 2087us) server_negotiation.cc:407] Connection header received
0928 10:44:51.772289 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:51.772294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:51.772372 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:51.772462 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:51.775163 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.776260 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.778788 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.779404 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.779505 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.785582 (+ 6077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:51.785591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:51.785596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:51.785659 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:51.789517 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.790873 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.790880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.790885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.791067 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.791734 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.791740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.791744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.792160 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:51.792291 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:51.793409 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0928 10:44:51.793704 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":1220,"threads_started":1}
I20260928 10:44:51.794101 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.767022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:54337 (local address 127.8.218.252:45803)
0928 10:44:51.767903 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:51.767907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:51.770418 (+ 2511us) server_negotiation.cc:407] Connection header received
0928 10:44:51.772195 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:51.772201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:51.772283 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:51.772379 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:51.775131 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.776169 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.778771 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.779325 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.779407 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.786095 (+ 6688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:51.786103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:51.786108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:51.786151 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:51.789664 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.790874 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.790880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.790885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.791064 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.791734 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.791740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.791744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.792125 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:51.792267 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:51.793432 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0928 10:44:51.793716 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":597,"threads_started":1}
Sep 28 10:44:51 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.195@KRBTEST.COM
I20260928 10:44:51.795706 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.766966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:36459 (local address 127.8.218.254:38155)
0928 10:44:51.767485 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:51.767492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:51.770270 (+ 2778us) server_negotiation.cc:407] Connection header received
0928 10:44:51.772666 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:51.772672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:51.772749 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:51.772864 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:51.775986 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.777090 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.778962 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.779628 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.779750 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.788066 (+ 8316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:51.788074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:51.788080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:51.788148 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:51.791828 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.793037 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.793044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.793048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.793293 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.793871 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.793877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.793880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.794230 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:51.794906 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:51.795056 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 10:44:51.795312 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260928 10:44:51.796736 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.768902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:45803 (local address 127.8.218.194:54337)
0928 10:44:51.769682 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:51.770264 (+ 582us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:51.771859 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:51.772692 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:51.772696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:51.773681 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:51.774884 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.774919 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.776339 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.776345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.778557 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.778576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.779734 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.779741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.780154 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.781146 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:51.781186 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:51.785836 (+ 4650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:51.789805 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.789809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.789813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.790703 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.791230 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.791235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.791239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.791591 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.793017 (+ 1426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:51.793022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:51.793166 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 10:44:51.794415 (+ 1249us) client_negotiation.cc:240] Negotiation successful
0928 10:44:51.795207 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":522,"spinlock_wait_cycles":84736,"thread_start_us":773,"threads_started":1}
I20260928 10:44:51.797257 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.768279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.194:36459)
0928 10:44:51.769682 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:51.769756 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:51.771859 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:51.772902 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:51.772906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:51.773781 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:51.774884 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.774919 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.777281 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.777287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.778841 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.778854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.779828 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.779834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.780181 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.781146 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:51.781187 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:51.784876 (+ 3689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:51.792338 (+ 7462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.792343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.792347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.792861 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.793546 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.793551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.793555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.793721 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.794375 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:51.794380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:51.794552 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 10:44:51.794676 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 10:44:51.795077 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":204,"spinlock_wait_cycles":32000,"thread_start_us":674,"threads_started":1}
I20260928 10:44:51.798043 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.769148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.8.218.194:35861)
0928 10:44:51.770864 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:51.770885 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:51.771864 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:51.772657 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:51.772669 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:51.773821 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:51.774892 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.774921 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.776487 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.776493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.778554 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:51.778576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.779870 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.779880 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:51.780135 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.781337 (+ 1202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:51.781360 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:51.785293 (+ 3933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:51.789742 (+ 4449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.789752 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.789774 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.790703 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.791230 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:51.791236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:51.791239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:51.791591 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:51.792374 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:51.792383 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:51.792893 (+ 510us) client_negotiation.cc:769] Sending connection context
0928 10:44:51.794769 (+ 1876us) client_negotiation.cc:240] Negotiation successful
0928 10:44:51.797699 (+ 2930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1601,"mutex_wait_us":178,"spinlock_wait_cycles":50560,"thread_start_us":1430,"threads_started":1}
I20260928 10:44:51.799142 16355 heartbeater.cc:347] Connected to a master server at 127.8.218.253:40141
I20260928 10:44:51.799220 16353 heartbeater.cc:347] Connected to a master server at 127.8.218.252:45803
I20260928 10:44:51.799258 16354 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38155
I20260928 10:44:51.799525 16354 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:51.799530 16355 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:51.799531 16353 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:51.800850 16354 heartbeater.cc:511] Master 127.8.218.254:38155 requested a full tablet report, sending...
I20260928 10:44:51.800933 16353 heartbeater.cc:511] Master 127.8.218.252:45803 requested a full tablet report, sending...
I20260928 10:44:51.800971 16355 heartbeater.cc:511] Master 127.8.218.253:40141 requested a full tablet report, sending...
I20260928 10:44:51.802816 15232 ts_manager.cc:194] Re-registered known tserver with Master: 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733)
I20260928 10:44:51.802964 16029 ts_manager.cc:194] Registered new tserver with Master: 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733)
I20260928 10:44:51.803000 15833 ts_manager.cc:194] Registered new tserver with Master: 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733)
I20260928 10:44:51.804028 15232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:36459
I20260928 10:44:51.804858 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:51.775335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.8.218.195:33793)
0928 10:44:51.776023 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:51.776032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:51.781583 (+ 5551us) server_negotiation.cc:407] Connection header received
0928 10:44:51.782074 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:51.782087 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:51.782430 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:51.782587 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:51.785024 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.786661 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.788701 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:51.789792 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:51.789947 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:51.795971 (+ 6024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:51.795995 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:51.796016 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:51.796066 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:51.800570 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.801726 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.801737 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.801741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.801974 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:51.802897 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:51.802903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:51.802906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:51.803437 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:51.803741 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:51.804117 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 10:44:51.804459 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":342,"threads_started":1}
I20260928 10:44:51.806836 15685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:51.891196 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 15562
I20260928 10:44:51.904868 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:33793
--local_ip_for_outbound_sockets=127.8.218.195
--tserver_master_addrs=127.8.218.254:38155
--webserver_port=36645
--webserver_interface=127.8.218.195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab
--principal=oryx/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141,127.8.218.252:45803 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:44:52.190172 16208 heartbeater.cc:503] Master 127.8.218.254:38155 was elected leader, sending a full tablet report...
W20260928 10:44:52.257311 16366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:52.257783 16366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:52.257937 16366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:52.265796 16366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:52.265929 16366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:52.266052 16366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:52.276571 16366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab
--principal=oryx/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:33793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=36645
--tserver_master_addrs=127.8.218.254:38155,127.8.218.253:40141,127.8.218.252:45803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:52.278292 16366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:52.279798 16366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:52.295049 16371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:52.295089 16374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:52.295135 16372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:52.296329 16366 server_base.cc:1034] running on GCE node
I20260928 10:44:52.297051 16366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:52.298722 16366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:52.300056 16366 hybrid_clock.cc:648] HybridClock initialized: now 1790592292299984 us; error 60 us; skew 500 ppm
I20260928 10:44:52.300482 16366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:52 dist-test-slave-mn38 krb5kdc[15188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592292, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:52.304278 16366 init.cc:377] Logged in from keytab as oryx/127.8.218.195@KRBTEST.COM (short username oryx)
I20260928 10:44:52.306073 16366 webserver.cc:460] Webserver started at http://127.8.218.195:36645/ using document root <none> and password file <none>
I20260928 10:44:52.306851 16366 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:52.307005 16366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:52.313021 16366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 10:44:52.316325 16381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:52.318179 16366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:52.318397 16366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "251b35975dd6493f99a3a2d66bdcf20b"
format_stamp: "Formatted at 2026-09-28 10:44:48 on dist-test-slave-mn38"
server_key: "2c629a3db368e2eaa2fe31d7438dcdb1"
server_key_iv: "94c411f649149885445bb6ff0a67d80e"
server_key_version: "encryptionkey@0"
I20260928 10:44:52.319519 16366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:52.337081 16366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:52.341800 16366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:52.342059 16366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:52.343101 16366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:44:52.344591 16366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:52.344687 16366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:52.344772 16366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:52.344828 16366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:52.376748 16366 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:33793
I20260928 10:44:52.376811 16493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:33793 every 8 connection(s)
I20260928 10:44:52.378278 16366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:52.387058 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 16366
Sep 28 10:44:52 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592292, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
Sep 28 10:44:52 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592292, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.252@KRBTEST.COM
Sep 28 10:44:52 dist-test-slave-mn38 krb5kdc[15188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592292, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.253@KRBTEST.COM
I20260928 10:44:52.433423 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.381232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:36177 (local address 127.8.218.254:38155)
0928 10:44:52.394150 (+ 12918us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.394157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.397248 (+ 3091us) server_negotiation.cc:407] Connection header received
0928 10:44:52.400584 (+ 3336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.400590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.400680 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.400810 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.404278 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.405892 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.408782 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.409547 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.409701 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.416358 (+ 6657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:52.416368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:52.416374 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:52.416429 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:52.421108 (+ 4679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.429917 (+ 8809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.429924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.429928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.430126 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.430770 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.430775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.430778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.431179 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:52.431333 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.432655 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.433018 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12800,"thread_start_us":12693,"threads_started":1}
I20260928 10:44:52.434823 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.381957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.195:36177)
0928 10:44:52.397127 (+ 15170us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:52.397149 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:52.400049 (+ 2900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:52.401299 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:52.401314 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:52.402959 (+ 1645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:52.403866 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.403903 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.406125 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.406131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.408563 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.408589 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.409811 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.409819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.410137 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.411051 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:52.411085 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:52.415989 (+ 4904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:52.421429 (+ 5440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:52.421477 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:52.421504 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:52.429720 (+ 8216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:52.430295 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:52.430300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:52.430304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:52.430623 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:52.431425 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:52.431436 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:52.431987 (+ 551us) client_negotiation.cc:769] Sending connection context
0928 10:44:52.432405 (+ 418us) client_negotiation.cc:240] Negotiation successful
0928 10:44:52.433066 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14990,"mutex_wait_us":86,"spinlock_wait_cycles":72704,"thread_start_us":15014,"threads_started":1}
I20260928 10:44:52.436694 16495 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38155
I20260928 10:44:52.437059 16495 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:52.438050 16495 heartbeater.cc:511] Master 127.8.218.254:38155 requested a full tablet report, sending...
I20260928 10:44:52.439908 15232 ts_manager.cc:194] Re-registered known tserver with Master: 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793)
I20260928 10:44:52.440397 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.382530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:50295 (local address 127.8.218.252:45803)
0928 10:44:52.394837 (+ 12307us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.394845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.396515 (+ 1670us) server_negotiation.cc:407] Connection header received
0928 10:44:52.400566 (+ 4051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.400571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.400661 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.400778 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.405553 (+ 4775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.406714 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.410910 (+ 4196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.411552 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.411665 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.429860 (+ 18195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:52.429869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:52.429874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:52.429921 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:52.433528 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.436135 (+ 2607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.436141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.436145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.436329 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.437695 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.437702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.437704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.438081 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:52.439219 (+ 1138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.439803 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.440028 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12183,"thread_start_us":12093,"threads_started":1}
I20260928 10:44:52.440447 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.381957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:45803 (local address 127.8.218.195:50295)
0928 10:44:52.396209 (+ 14252us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:52.396289 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:52.400043 (+ 3754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:52.401298 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:52.401314 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:52.402544 (+ 1230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:52.405002 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.405019 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.407038 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.407042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.409963 (+ 2921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.409978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.411998 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.412005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.412178 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.413553 (+ 1375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:52.413579 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:52.428884 (+ 15305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:52.433870 (+ 4986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:52.433876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:52.433879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:52.434424 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:52.436539 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:52.436544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:52.436547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:52.436799 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:52.439370 (+ 2571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:52.439374 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:52.439536 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 10:44:52.439656 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 10:44:52.439990 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13825,"mutex_wait_us":2504,"spinlock_wait_cycles":36864,"thread_start_us":16522,"threads_started":1}
I20260928 10:44:52.441150 15232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:36177
I20260928 10:44:52.441461 16494 heartbeater.cc:347] Connected to a master server at 127.8.218.252:45803
I20260928 10:44:52.441599 16494 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:52.442068 16494 heartbeater.cc:511] Master 127.8.218.252:45803 requested a full tablet report, sending...
I20260928 10:44:52.442332 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.381249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:35925 (local address 127.8.218.253:40141)
0928 10:44:52.393788 (+ 12539us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.393796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.399146 (+ 5350us) server_negotiation.cc:407] Connection header received
0928 10:44:52.400485 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.400491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.400605 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.400694 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.404641 (+ 3947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.405908 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.414088 (+ 8180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.414797 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.414930 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.435227 (+ 20297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:52.435235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:52.435240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:52.435290 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:52.439053 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.439946 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.439953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.439956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.440122 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.440594 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.440600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.440603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.440971 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:52.441104 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.441688 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.441939 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12375,"thread_start_us":12910,"threads_started":1}
I20260928 10:44:52.443390 16029 ts_manager.cc:194] Registered new tserver with Master: 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793)
I20260928 10:44:52.443555 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.381971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.8.218.195:35925)
0928 10:44:52.398994 (+ 17023us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:52.399016 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:52.400045 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:52.401298 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:52.401314 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:52.403313 (+ 1999us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:52.404246 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.404263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.408835 (+ 4572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.408839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.413269 (+ 4430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.413290 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.416427 (+ 3137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.416433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.416676 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.417530 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:52.417564 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:52.434958 (+ 17394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:52.439259 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:52.439264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:52.439268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:52.439773 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:52.440279 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:52.440284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:52.440287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:52.440451 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:52.441190 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:52.441195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:52.441347 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 10:44:52.442621 (+ 1274us) client_negotiation.cc:240] Negotiation successful
0928 10:44:52.443063 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16859,"mutex_wait_us":358,"thread_start_us":13317,"threads_started":1}
I20260928 10:44:52.444506 16496 heartbeater.cc:347] Connected to a master server at 127.8.218.253:40141
I20260928 10:44:52.444659 16496 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:52.445147 16496 heartbeater.cc:511] Master 127.8.218.253:40141 requested a full tablet report, sending...
I20260928 10:44:52.446472 15833 ts_manager.cc:194] Registered new tserver with Master: 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793)
I20260928 10:44:52.461570 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.401644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.8.218.254:38155)
0928 10:44:52.433739 (+ 32095us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.433744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.433765 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:44:52.433823 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.433828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.433921 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.434030 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.446093 (+ 12063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.447264 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.449357 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.450061 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.450210 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.453811 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:52.453818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:52.453823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:52.453872 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:52.456814 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.457918 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.457924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.457928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.458104 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.459145 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.459221 (+ 76us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.459226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.459564 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:52.459844 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.460745 (+ 901us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.461030 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31983,"thread_start_us":45018,"threads_started":1}
I20260928 10:44:52.462559 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.402615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.8.218.252:45803)
0928 10:44:52.440712 (+ 38097us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.440718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.440753 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:44:52.440800 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.440805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.440897 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.440989 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.447491 (+ 6502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.448579 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.450447 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.450966 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.451098 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.454361 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:52.454368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:52.454373 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:52.454416 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:52.457519 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.458268 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.458272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.458275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.458401 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.458797 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.458802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.458805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.459173 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:52.459478 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.461915 (+ 2437us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.462198 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37987,"thread_start_us":44686,"threads_started":1}
I20260928 10:44:52.465070 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.418509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.8.218.253:40141)
0928 10:44:52.442575 (+ 24066us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.442580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.446521 (+ 3941us) server_negotiation.cc:407] Connection header received
0928 10:44:52.447361 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.447367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.447469 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.447571 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.450071 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.451253 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.453231 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.453916 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.454051 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.457164 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:52.457171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:52.457175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:52.457219 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:52.460643 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.461300 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.461306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.461310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.461493 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.461859 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.461865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.461868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.462257 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:52.462547 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.464377 (+ 1830us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.464680 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23959,"thread_start_us":27256,"threads_started":1}
I20260928 10:44:52.467556 15232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:52.475183 15232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:52.512925 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.499505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:41733 (local address 127.0.0.1:38878)
0928 10:44:52.500002 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:52.500017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:52.500128 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:52.505583 (+ 5455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:52.505589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:52.505632 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:52.506013 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.506028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.508022 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.508029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.510062 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.510079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.512070 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.512077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.512263 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.512303 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:44:52.512400 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 10:44:52.512546 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":304,"threads_started":1}
I20260928 10:44:52.513886 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.499829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.8.218.194:41733)
0928 10:44:52.500570 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.500580 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.500612 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:44:52.500733 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.500764 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.501098 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.501344 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:52.506267 (+ 4923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.507827 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.510234 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.511529 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.511680 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.511938 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.513354 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.513591 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":850,"threads_started":1}
I20260928 10:44:52.513957 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.498602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:33805 (local address 127.0.0.1:38086)
0928 10:44:52.499965 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:52.499991 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:52.500141 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:52.506850 (+ 6709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:52.506856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:52.506884 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:52.507216 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.507227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.509244 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.509249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.511215 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.511228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.513118 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.513124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.513294 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.513326 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:44:52.513422 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 10:44:52.513600 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"spinlock_wait_cycles":18304,"thread_start_us":1519,"threads_started":1}
I20260928 10:44:52.514217 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.498918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.8.218.193:33805)
0928 10:44:52.506075 (+ 7157us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.506085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.506116 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:44:52.506221 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.506230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.506564 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.506853 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:52.507394 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.509069 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.511379 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.512817 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.512939 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.513160 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.513516 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.513673 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6965,"thread_start_us":6892,"threads_started":1}
I20260928 10:44:52.515637 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.500485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33793 (local address 127.0.0.1:60350)
0928 10:44:52.502132 (+ 1647us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:52.502152 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:52.502273 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:52.508156 (+ 5883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:52.508161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:52.508191 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:52.508599 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.508609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.510786 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.510792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.512775 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.512787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.514965 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.514972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.515148 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.515180 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:44:52.515265 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 10:44:52.515387 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"thread_start_us":1267,"threads_started":1}
I20260928 10:44:52.516676 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.500831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.8.218.195:33793)
0928 10:44:52.506697 (+ 5866us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.506706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.506737 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:44:52.507638 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.507647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.507922 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.508779 (+ 857us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:52.509016 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.510631 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.513261 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.514748 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.514860 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.515027 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.516157 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.516351 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5693,"thread_start_us":5515,"threads_started":1}
I20260928 10:44:52.518373 16288 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0111645f2f47e0aa9f279263baedbc (DEFAULT_TABLE table=test-table [id=d8092ef7962948d283c667a57574d9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:52.518728 16142 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0111645f2f47e0aa9f279263baedbc (DEFAULT_TABLE table=test-table [id=d8092ef7962948d283c667a57574d9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:52.519902 16142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0111645f2f47e0aa9f279263baedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:52.519917 16288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0111645f2f47e0aa9f279263baedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:52.520287 16429 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0111645f2f47e0aa9f279263baedbc (DEFAULT_TABLE table=test-table [id=d8092ef7962948d283c667a57574d9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:52.521831 16429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0111645f2f47e0aa9f279263baedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:52.533576 16531 tablet_bootstrap.cc:492] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Bootstrap starting.
I20260928 10:44:52.537031 16531 tablet_bootstrap.cc:654] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:52.538746 16531 log.cc:824] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:52.541671 16531 tablet_bootstrap.cc:492] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: No bootstrap required, opened a new log
I20260928 10:44:52.541986 16531 ts_tablet_manager.cc:1397] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 10:44:52.542220 16533 tablet_bootstrap.cc:492] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Bootstrap starting.
I20260928 10:44:52.546195 16533 tablet_bootstrap.cc:654] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:52.546631 16532 tablet_bootstrap.cc:492] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: Bootstrap starting.
I20260928 10:44:52.548743 16533 log.cc:824] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:52.550559 16532 tablet_bootstrap.cc:654] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:52.551488 16531 raft_consensus.cc:348] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.551870 16531 raft_consensus.cc:374] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:52.551851 16532 log.cc:824] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:44:52.551967 16531 raft_consensus.cc:729] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fd0e6225a848f88b069315eb44875a, State: Initialized, Role: FOLLOWER
I20260928 10:44:52.552306 16533 tablet_bootstrap.cc:492] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: No bootstrap required, opened a new log
I20260928 10:44:52.552610 16533 ts_tablet_manager.cc:1397] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260928 10:44:52.552600 16531 consensus_queue.cc:260] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.554188 16531 ts_tablet_manager.cc:1428] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 10:44:52.555231 16532 tablet_bootstrap.cc:492] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: No bootstrap required, opened a new log
I20260928 10:44:52.555524 16532 ts_tablet_manager.cc:1397] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20260928 10:44:52.560817 16537 raft_consensus.cc:482] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:52.561174 16537 raft_consensus.cc:504] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.563755 16533 raft_consensus.cc:348] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.564316 16533 raft_consensus.cc:374] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:52.564435 16533 raft_consensus.cc:729] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251b35975dd6493f99a3a2d66bdcf20b, State: Initialized, Role: FOLLOWER
I20260928 10:44:52.565322 16533 consensus_queue.cc:260] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.570194 16537 leader_election.cc:288] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793), 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733)
I20260928 10:44:52.572712 16532 raft_consensus.cc:348] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.573535 16532 raft_consensus.cc:374] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:52.573729 16532 raft_consensus.cc:729] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a6c2fd3ad94007b848eb02279fd43d, State: Initialized, Role: FOLLOWER
I20260928 10:44:52.573870 16533 ts_tablet_manager.cc:1428] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Time spent starting tablet: real 0.021s user 0.021s sys 0.002s
I20260928 10:44:52.574787 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.564228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:59105 (local address 127.8.218.194:41733)
0928 10:44:52.564429 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.564434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.564454 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:44:52.564724 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.564730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.564818 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.564906 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:52.566085 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.567441 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.571034 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.572416 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.572560 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.572642 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.574357 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.574522 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:44:52.574735 16532 consensus_queue.cc:260] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.576411 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.563561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:41733 (local address 127.8.218.193:59105)
0928 10:44:52.564343 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:52.564371 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:52.564532 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:52.565125 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:52.565129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:52.565160 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:52.565860 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.565879 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.567721 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.567728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.570638 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.570669 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.573813 (+ 3144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.573821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.574035 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.574075 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:44:52.575300 (+ 1225us) client_negotiation.cc:240] Negotiation successful
0928 10:44:52.575499 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":1052,"threads_started":1}
I20260928 10:44:52.577389 16532 ts_tablet_manager.cc:1428] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: Time spent starting tablet: real 0.022s user 0.017s sys 0.000s
I20260928 10:44:52.577889 16495 heartbeater.cc:503] Master 127.8.218.254:38155 was elected leader, sending a full tablet report...
I20260928 10:44:52.578460 16354 heartbeater.cc:503] Master 127.8.218.254:38155 was elected leader, sending a full tablet report...
I20260928 10:44:52.580729 16308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0111645f2f47e0aa9f279263baedbc" candidate_uuid: "e3fd0e6225a848f88b069315eb44875a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a6c2fd3ad94007b848eb02279fd43d" is_pre_election: true
I20260928 10:44:52.581338 16308 raft_consensus.cc:2454] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fd0e6225a848f88b069315eb44875a in term 0.
I20260928 10:44:52.583822 16098 leader_election.cc:302] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56a6c2fd3ad94007b848eb02279fd43d, e3fd0e6225a848f88b069315eb44875a; no voters:
I20260928 10:44:52.584339 16537 raft_consensus.cc:2781] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:52.584489 16537 raft_consensus.cc:482] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:52.584583 16537 raft_consensus.cc:3037] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:52.586193 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.563880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:40703 (local address 127.8.218.195:33793)
0928 10:44:52.564112 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.564119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.576204 (+ 12085us) server_negotiation.cc:407] Connection header received
0928 10:44:52.578603 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.578608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.578684 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.578780 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:52.580117 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.581183 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.583727 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.584972 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.585113 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.585168 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.585801 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.585942 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:44:52.586730 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.563561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33793 (local address 127.8.218.193:40703)
0928 10:44:52.576102 (+ 12541us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:52.576123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:52.576273 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:52.579447 (+ 3174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:52.579452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:52.579485 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:52.579819 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.579834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.581473 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.581478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.583556 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:52.583575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.585345 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.585365 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:52.585602 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.585639 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 10:44:52.585742 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 10:44:52.585966 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12062,"mutex_wait_us":910,"thread_start_us":10594,"threads_started":1}
I20260928 10:44:52.587808 16537 raft_consensus.cc:504] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.587805 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0111645f2f47e0aa9f279263baedbc" candidate_uuid: "e3fd0e6225a848f88b069315eb44875a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251b35975dd6493f99a3a2d66bdcf20b" is_pre_election: true
I20260928 10:44:52.588392 16449 raft_consensus.cc:2454] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fd0e6225a848f88b069315eb44875a in term 0.
I20260928 10:44:52.588900 16448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0111645f2f47e0aa9f279263baedbc" candidate_uuid: "e3fd0e6225a848f88b069315eb44875a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251b35975dd6493f99a3a2d66bdcf20b"
I20260928 10:44:52.589056 16537 leader_election.cc:288] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [CANDIDATE]: Term 1 election: Requested vote from peers 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793), 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733)
I20260928 10:44:52.589169 16448 raft_consensus.cc:3037] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:52.589192 16308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0111645f2f47e0aa9f279263baedbc" candidate_uuid: "e3fd0e6225a848f88b069315eb44875a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a6c2fd3ad94007b848eb02279fd43d"
I20260928 10:44:52.589413 16308 raft_consensus.cc:3037] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:52.591828 16308 raft_consensus.cc:2454] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fd0e6225a848f88b069315eb44875a in term 1.
I20260928 10:44:52.591835 16448 raft_consensus.cc:2454] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fd0e6225a848f88b069315eb44875a in term 1.
I20260928 10:44:52.592396 16096 leader_election.cc:302] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 251b35975dd6493f99a3a2d66bdcf20b, 56a6c2fd3ad94007b848eb02279fd43d, e3fd0e6225a848f88b069315eb44875a; no voters:
I20260928 10:44:52.592798 16537 raft_consensus.cc:2781] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:52.593228 16537 raft_consensus.cc:686] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 1 LEADER]: Becoming Leader. State: Replica: e3fd0e6225a848f88b069315eb44875a, State: Running, Role: LEADER
I20260928 10:44:52.593983 16537 consensus_queue.cc:237] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } }
I20260928 10:44:52.599341 15233 catalog_manager.cc:5495] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fd0e6225a848f88b069315eb44875a (127.8.218.193). New cstate: current_term: 1 leader_uuid: "e3fd0e6225a848f88b069315eb44875a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } health_report { overall_health: UNKNOWN } } }
I20260928 10:44:52.622901 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.617138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.8.218.193:33805)
0928 10:44:52.617362 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.617369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.617389 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:44:52.617648 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.617652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.617735 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.617876 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:52.618308 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.619057 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.620557 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.621017 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.621134 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.621722 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:52.622276 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:52.622436 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.622540 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.622633 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 10:44:52.629416 16210 tablet.cc:2369] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:44:52.631263 16497 tablet.cc:2369] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:44:52.632465 16448 raft_consensus.cc:1261] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 1 FOLLOWER]: Refusing update from remote peer e3fd0e6225a848f88b069315eb44875a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:52.632472 16308 raft_consensus.cc:1261] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 1 FOLLOWER]: Refusing update from remote peer e3fd0e6225a848f88b069315eb44875a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:52.633584 16542 consensus_queue.cc:1035] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [LEADER]: Connected to new peer: Peer: permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:52.633965 16537 consensus_queue.cc:1035] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:52.648985 16545 mvcc.cc:205] Tried to move back new op lower bound from 7334266030610964480 to 7334266030468988928. Current Snapshot: MvccSnapshot[applied={T|T < 7334266030610964480}]
I20260928 10:44:52.650586 16546 mvcc.cc:205] Tried to move back new op lower bound from 7334266030610964480 to 7334266030468988928. Current Snapshot: MvccSnapshot[applied={T|T < 7334266030610964480}]
I20260928 10:44:52.670749 15233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:52.671078 15233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:52.682061 15233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c02e72f8941457a9bf91b0c1f0a0081: Sending DeleteTablet for 3 replicas of tablet 8e0111645f2f47e0aa9f279263baedbc
I20260928 10:44:52.683363 16288 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e0111645f2f47e0aa9f279263baedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:52 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:38878
I20260928 10:44:52.683715 16429 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e0111645f2f47e0aa9f279263baedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:52 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:60350
I20260928 10:44:52.683771 16142 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e0111645f2f47e0aa9f279263baedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:52 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:38086
I20260928 10:44:52.697418 16557 tablet_replica.cc:330] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: stopping tablet replica
I20260928 10:44:52.698302 16557 raft_consensus.cc:2229] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:52.701871 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 15201
I20260928 10:44:52.703507 15263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:52.704840 16556 tablet_replica.cc:330] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: stopping tablet replica
I20260928 10:44:52.700335 16557 raft_consensus.cc:2258] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:52.705416 16556 raft_consensus.cc:2229] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:52.705899 16555 tablet_replica.cc:330] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: stopping tablet replica
I20260928 10:44:52.706110 16556 raft_consensus.cc:2258] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:52.706849 16555 raft_consensus.cc:2229] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:52.709544 16555 raft_consensus.cc:2258] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:52.711362 16556 ts_tablet_manager.cc:1905] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:52.712968 16557 ts_tablet_manager.cc:1905] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:52.715001 16555 ts_tablet_manager.cc:1905] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:52.721922 16557 ts_tablet_manager.cc:1918] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:52.722160 16557 log.cc:1196] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/8e0111645f2f47e0aa9f279263baedbc
I20260928 10:44:52.722798 16557 ts_tablet_manager.cc:1939] T 8e0111645f2f47e0aa9f279263baedbc P e3fd0e6225a848f88b069315eb44875a: Deleting consensus metadata
I20260928 10:44:52.724397 16556 ts_tablet_manager.cc:1918] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:52.726437 16556 log.cc:1196] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/8e0111645f2f47e0aa9f279263baedbc
I20260928 10:44:52.727777 16556 ts_tablet_manager.cc:1939] T 8e0111645f2f47e0aa9f279263baedbc P 251b35975dd6493f99a3a2d66bdcf20b: Deleting consensus metadata
I20260928 10:44:52.735221 16555 ts_tablet_manager.cc:1918] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:52.735695 16555 log.cc:1196] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/8e0111645f2f47e0aa9f279263baedbc
I20260928 10:44:52.736577 16555 ts_tablet_manager.cc:1939] T 8e0111645f2f47e0aa9f279263baedbc P 56a6c2fd3ad94007b848eb02279fd43d: Deleting consensus metadata
I20260928 10:44:52.740655 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.686208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.8.218.253:40141)
0928 10:44:52.686406 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.686413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.703340 (+ 16927us) server_negotiation.cc:407] Connection header received
0928 10:44:52.703631 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.703637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.703754 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.703880 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.709045 (+ 5165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.710559 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.714786 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.715525 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.715689 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.731314 (+ 15625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:52.731323 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:52.731328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:52.731382 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:52.735310 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.736388 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.736395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.736398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.736684 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.737699 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.737705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.737709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.738121 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:52.738381 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.739999 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.740265 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:44:52.759641 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.700218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.8.218.252:45803)
0928 10:44:52.700552 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.700559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.739792 (+ 39233us) server_negotiation.cc:407] Connection header received
0928 10:44:52.741142 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.741149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.741238 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.741312 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.743363 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.744590 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.746669 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.747333 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:52.747450 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:52.751442 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:52.751452 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:52.751457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:52.751516 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:52.755397 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.756608 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.756616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.756620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.756800 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:52.757277 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:52.757283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:52.757287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:52.757764 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:52.758017 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:52.758841 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 10:44:52.759148 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:44:52.939757 15218 catalog_manager.cc:4841] TS 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793): tablet 8e0111645f2f47e0aa9f279263baedbc (table test-table [id=d8092ef7962948d283c667a57574d9e3]) successfully deleted
I20260928 10:44:52.940155 15218 catalog_manager.cc:4841] TS e3fd0e6225a848f88b069315eb44875a (127.8.218.193:33805): tablet 8e0111645f2f47e0aa9f279263baedbc (table test-table [id=d8092ef7962948d283c667a57574d9e3]) successfully deleted
I20260928 10:44:52.940357 15220 catalog_manager.cc:4841] TS 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733): tablet 8e0111645f2f47e0aa9f279263baedbc (table test-table [id=d8092ef7962948d283c667a57574d9e3]) successfully deleted
I20260928 10:44:52.940603 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 15201
I20260928 10:44:53.017083 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.980848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.8.218.252:45803)
0928 10:44:52.981087 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.981094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.990836 (+ 9742us) server_negotiation.cc:407] Connection header received
0928 10:44:52.991169 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.991177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.991275 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.991355 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:52.995575 (+ 4220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:52.997248 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:53.000503 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:53.001317 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:53.001501 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:53.007530 (+ 6029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:53.007539 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:53.007543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:53.007589 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:53.012227 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:53.013421 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:53.013428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:53.013433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:53.013652 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:53.014147 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:53.014154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:53.014157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:53.014587 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:53.014837 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:53.016283 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0928 10:44:53.016711 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 10:44:53.020272 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:52.987219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.8.218.253:40141)
0928 10:44:52.987469 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:52.987476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:52.992337 (+ 4861us) server_negotiation.cc:407] Connection header received
0928 10:44:52.993502 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:52.993509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:52.993617 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:52.993692 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:53.002433 (+ 8741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:53.003322 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:53.006363 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:53.007047 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:53.007214 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:53.010936 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:53.010945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:53.010951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:53.011003 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:53.014803 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:53.016853 (+ 2050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:53.016861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:53.016865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:53.017078 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:53.018746 (+ 1668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:53.018753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:53.018757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:53.019080 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:53.019317 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:53.019700 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 10:44:53.019941 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 10:44:53.663697 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:53.662396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.193:50227)
0928 10:44:53.663301 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:53.663453 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":778,"threads_started":1}
W20260928 10:44:53.664469 16208 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:38155 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
I20260928 10:44:53.666256 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:53.663729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.194:46283)
0928 10:44:53.665892 (+ 2163us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:53.666046 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2039,"thread_start_us":2204,"threads_started":1}
W20260928 10:44:53.666692 16354 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:38155 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
I20260928 10:44:53.667661 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:53.664835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.195:55177)
0928 10:44:53.667325 (+ 2490us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:53.667459 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2369,"thread_start_us":2273,"threads_started":1}
W20260928 10:44:53.668090 16495 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:38155 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
I20260928 10:44:54.181205 16570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c02e72f8941457a9bf91b0c1f0a0081)
I20260928 10:44:54.181527 16571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c02e72f8941457a9bf91b0c1f0a0081)
I20260928 10:44:54.181592 16570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } }
I20260928 10:44:54.181761 16571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } }
I20260928 10:44:54.183131 16571 leader_election.cc:288] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c02e72f8941457a9bf91b0c1f0a0081 (127.8.218.254:38155), 74a949e40a7743c89969c2db112fbb27 (127.8.218.252:45803)
I20260928 10:44:54.189356 16570 leader_election.cc:288] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c02e72f8941457a9bf91b0c1f0a0081 (127.8.218.254:38155), 6be0af7dde2947cd98e2fae61320ea19 (127.8.218.253:40141)
I20260928 10:44:54.197458 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.183330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54222)
0928 10:44:54.196920 (+ 13590us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.197153 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13477,"mutex_wait_us":51,"thread_start_us":13321,"threads_started":1}
W20260928 10:44:54.197913 15818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
I20260928 10:44:54.198127 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.183250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54206)
0928 10:44:54.197349 (+ 14099us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.197727 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13963,"thread_start_us":13859,"threads_started":1}
W20260928 10:44:54.198503 16014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
I20260928 10:44:54.208153 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.207609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54226)
0928 10:44:54.207791 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.207907 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 10:44:54.208648 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.183330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:45803 (local address 127.0.0.1:46504)
0928 10:44:54.196215 (+ 12885us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.196246 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:54.196447 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:54.199523 (+ 3076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:54.199530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:54.199571 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:54.200008 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.200023 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.202387 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.202395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.205196 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.205234 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.207627 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.207633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.207827 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.207865 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:44:54.207962 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 10:44:54.208171 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12692,"mutex_wait_us":3,"thread_start_us":14317,"threads_started":1}
W20260928 10:44:54.208961 15818 leader_election.cc:334] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c02e72f8941457a9bf91b0c1f0a0081 (127.8.218.254:38155): Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
I20260928 10:44:54.209614 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.208680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54238)
0928 10:44:54.209114 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.209308 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277}
I20260928 10:44:54.209697 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.183670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.8.218.252:45803)
0928 10:44:54.198750 (+ 15080us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.198758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.198785 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:44:54.198878 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.198885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.198979 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.199110 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:54.200261 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.202049 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.205582 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.207456 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.207618 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.208602 (+ 984us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.208661 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.209163 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14855,"thread_start_us":14537,"threads_started":1}
W20260928 10:44:54.209949 16014 leader_election.cc:334] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c02e72f8941457a9bf91b0c1f0a0081 (127.8.218.254:38155): Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
I20260928 10:44:54.210405 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6be0af7dde2947cd98e2fae61320ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "74a949e40a7743c89969c2db112fbb27" is_pre_election: true
I20260928 10:44:54.210901 16039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6be0af7dde2947cd98e2fae61320ea19 in term 1.
I20260928 10:44:54.211644 15817 leader_election.cc:302] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6be0af7dde2947cd98e2fae61320ea19, 74a949e40a7743c89969c2db112fbb27; no voters: 4c02e72f8941457a9bf91b0c1f0a0081
I20260928 10:44:54.212119 16571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 10:44:54.212258 16571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c02e72f8941457a9bf91b0c1f0a0081)
I20260928 10:44:54.212360 16571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:44:54.215479 16571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } }
I20260928 10:44:54.216624 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.199018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:40141 (local address 127.0.0.1:48112)
0928 10:44:54.200080 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.200344 (+ 264us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:54.200517 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:54.207368 (+ 6851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:54.207374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:54.207426 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:54.207746 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.207760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.209520 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.209526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.211632 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.211655 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.215774 (+ 4119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.215782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.216014 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.216042 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 10:44:54.216144 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 10:44:54.216308 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":5407,"threads_started":1}
I20260928 10:44:54.216723 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.198676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.8.218.253:40141)
0928 10:44:54.205577 (+ 6901us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.205584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.205618 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:44:54.205702 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.205710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.206763 (+ 1053us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.206903 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:54.207925 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.209319 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.211885 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.215544 (+ 3659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.215693 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.215820 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.216196 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.216341 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":6670,"thread_start_us":7133,"threads_started":1}
I20260928 10:44:54.217424 15842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74a949e40a7743c89969c2db112fbb27" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6be0af7dde2947cd98e2fae61320ea19" is_pre_election: true
I20260928 10:44:54.217569 16571 leader_election.cc:288] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [CANDIDATE]: Term 2 election: Requested vote from peers 4c02e72f8941457a9bf91b0c1f0a0081 (127.8.218.254:38155), 74a949e40a7743c89969c2db112fbb27 (127.8.218.252:45803)
I20260928 10:44:54.217921 15842 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a949e40a7743c89969c2db112fbb27 in current term 2: Already voted for candidate 6be0af7dde2947cd98e2fae61320ea19 in this term.
I20260928 10:44:54.218125 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.217696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54256)
0928 10:44:54.217876 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.217983 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 10:44:54.218030 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6be0af7dde2947cd98e2fae61320ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "74a949e40a7743c89969c2db112fbb27"
I20260928 10:44:54.218294 16039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:44:54.218730 16016 leader_election.cc:302] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a949e40a7743c89969c2db112fbb27; no voters: 4c02e72f8941457a9bf91b0c1f0a0081, 6be0af7dde2947cd98e2fae61320ea19
I20260928 10:44:54.219255 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.218819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54266)
0928 10:44:54.218998 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.219101 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 10:44:54.219435 15818 leader_election.cc:334] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4c02e72f8941457a9bf91b0c1f0a0081 (127.8.218.254:38155): Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
I20260928 10:44:54.220816 16039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be0af7dde2947cd98e2fae61320ea19 in term 2.
I20260928 10:44:54.221019 16570 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 10:44:54.221275 15817 leader_election.cc:302] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6be0af7dde2947cd98e2fae61320ea19, 74a949e40a7743c89969c2db112fbb27; no voters: 4c02e72f8941457a9bf91b0c1f0a0081
I20260928 10:44:54.221627 16571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:44:54.222085 16571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [term 2 LEADER]: Becoming Leader. State: Replica: 6be0af7dde2947cd98e2fae61320ea19, State: Running, Role: LEADER
I20260928 10:44:54.222474 16571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } }
I20260928 10:44:54.224552 16580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6be0af7dde2947cd98e2fae61320ea19. Latest consensus state: current_term: 2 leader_uuid: "6be0af7dde2947cd98e2fae61320ea19" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:54.225018 16580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:54.225769 16582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:54.229597 16582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:54.230501 16582 catalog_manager.cc:1259] Loaded cluster ID: 982ffa990fe5464ba4040ba7b39b98a7
I20260928 10:44:54.230589 16582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:54.231762 16582 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:54.232613 16582 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Loaded TSK: 0
I20260928 10:44:54.233572 16582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:44:54.272733 15832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:44:54.280510 15832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:44:54.285427 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.284888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54280)
0928 10:44:54.285131 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.285254 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260928 10:44:54.285606 16039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [term 2 FOLLOWER]: Refusing update from remote peer 6be0af7dde2947cd98e2fae61320ea19: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 10:44:54.286582 16580 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 10:44:54.286901 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.286328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54296)
0928 10:44:54.286620 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.286731 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 10:44:54.287155 15818 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 -> Peer 4c02e72f8941457a9bf91b0c1f0a0081 (127.8.218.254:38155): Couldn't send request to peer 4c02e72f8941457a9bf91b0c1f0a0081. Status: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 10:44:54.289584 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6be0af7dde2947cd98e2fae61320ea19. Latest consensus state: current_term: 2 leader_uuid: "6be0af7dde2947cd98e2fae61320ea19" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:54.289891 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:44:54.291671 16571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6be0af7dde2947cd98e2fae61320ea19" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:54.292001 16571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19 [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:54.292580 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6be0af7dde2947cd98e2fae61320ea19" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4c02e72f8941457a9bf91b0c1f0a0081" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38155 } } peers { permanent_uuid: "6be0af7dde2947cd98e2fae61320ea19" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "74a949e40a7743c89969c2db112fbb27" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 45803 } attrs { promote: false } } }
I20260928 10:44:54.292955 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a949e40a7743c89969c2db112fbb27 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:44:54.311784 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.302796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:33805 (local address 127.0.0.1:48468)
0928 10:44:54.303058 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.303075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:54.303177 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:54.304447 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:54.304452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:54.304497 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:54.304834 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.304847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.306878 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.306887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.309309 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.309330 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.310931 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.310938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.311153 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.311193 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:44:54.311268 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 10:44:54.311431 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260928 10:44:54.313079 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.303131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.8.218.193:33805)
0928 10:44:54.303747 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.303755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.303781 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:44:54.304191 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.304197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.304283 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.305008 (+ 725us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:54.305210 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.306613 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.309658 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.310672 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.312376 (+ 1704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.312541 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.312601 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.312756 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":462,"threads_started":1}
I20260928 10:44:54.316610 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.302460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:41733 (local address 127.0.0.1:60290)
0928 10:44:54.302642 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.302663 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:54.302778 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:54.310329 (+ 7551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:54.310334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:54.310371 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:54.310558 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.310570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.311998 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.312004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.313770 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.313788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.315852 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.315859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.316036 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.316064 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 10:44:54.316148 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 10:44:54.316301 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 10:44:54.316666 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.302758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.8.218.194:41733)
0928 10:44:54.309925 (+ 7167us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.309932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.309958 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:44:54.310041 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.310047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.310148 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.310270 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:54.310768 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.311802 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.314337 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.315656 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.315788 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.315896 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.316198 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.316359 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7005,"thread_start_us":6876,"threads_started":1}
I20260928 10:44:54.316569 16142 tablet_service.cc:1467] Processing CreateTablet for tablet 232ce7a99ddc4c9b85adf160dd76bcd5 (DEFAULT_TABLE table=test-table [id=0c7e09922564480b9db3a1a26def7adf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:54.317096 16142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232ce7a99ddc4c9b85adf160dd76bcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:54.318845 16288 tablet_service.cc:1467] Processing CreateTablet for tablet 232ce7a99ddc4c9b85adf160dd76bcd5 (DEFAULT_TABLE table=test-table [id=0c7e09922564480b9db3a1a26def7adf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:54.319252 16288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232ce7a99ddc4c9b85adf160dd76bcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:54.320951 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.303215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:33793 (local address 127.0.0.1:53886)
0928 10:44:54.312063 (+ 8848us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.312084 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:54.312202 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:54.315213 (+ 3011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:54.315219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:54.315283 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:44:54.315546 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.315558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.317098 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.317104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.318371 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:54.318389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.320112 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.320120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:54.320343 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.320387 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 10:44:54.320471 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 10:44:54.320640 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8748,"thread_start_us":12358,"threads_started":1}
I20260928 10:44:54.321916 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.303454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.8.218.195:33793)
0928 10:44:54.314569 (+ 11115us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.314577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.314603 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:44:54.314744 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.314750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.314881 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.315713 (+ 832us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:44:54.315916 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.316931 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.318581 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.319828 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.321213 (+ 1385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.321336 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.321420 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.321621 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10953,"thread_start_us":11571,"threads_started":1}
I20260928 10:44:54.324208 16593 tablet_bootstrap.cc:492] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: Bootstrap starting.
I20260928 10:44:54.324435 16429 tablet_service.cc:1467] Processing CreateTablet for tablet 232ce7a99ddc4c9b85adf160dd76bcd5 (DEFAULT_TABLE table=test-table [id=0c7e09922564480b9db3a1a26def7adf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:44:54.324958 16429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232ce7a99ddc4c9b85adf160dd76bcd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:54.326042 16593 tablet_bootstrap.cc:654] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:54.326464 16594 tablet_bootstrap.cc:492] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: Bootstrap starting.
I20260928 10:44:54.328141 16594 tablet_bootstrap.cc:654] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:54.328416 16593 tablet_bootstrap.cc:492] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: No bootstrap required, opened a new log
I20260928 10:44:54.328579 16593 ts_tablet_manager.cc:1397] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:54.329473 16593 raft_consensus.cc:348] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.329748 16593 raft_consensus.cc:374] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:54.329838 16593 raft_consensus.cc:729] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fd0e6225a848f88b069315eb44875a, State: Initialized, Role: FOLLOWER
I20260928 10:44:54.330142 16593 consensus_queue.cc:260] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.335511 16594 tablet_bootstrap.cc:492] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: No bootstrap required, opened a new log
I20260928 10:44:54.335855 16594 ts_tablet_manager.cc:1397] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 10:44:54.336944 16594 raft_consensus.cc:348] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.337262 16594 raft_consensus.cc:374] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:54.337257 16593 ts_tablet_manager.cc:1428] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260928 10:44:54.337353 16594 raft_consensus.cc:729] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a6c2fd3ad94007b848eb02279fd43d, State: Initialized, Role: FOLLOWER
I20260928 10:44:54.337910 16598 tablet_bootstrap.cc:492] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: Bootstrap starting.
I20260928 10:44:54.338425 16594 consensus_queue.cc:260] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.339811 16598 tablet_bootstrap.cc:654] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:54.341616 16594 ts_tablet_manager.cc:1428] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260928 10:44:54.342562 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.341230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.193:35497)
0928 10:44:54.341969 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.342211 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":395,"threads_started":1}
I20260928 10:44:54.346300 16209 heartbeater.cc:503] Master 127.8.218.253:40141 was elected leader, sending a full tablet report...
I20260928 10:44:54.346544 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.345944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.193:51741)
0928 10:44:54.346171 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.346295 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 10:44:54.346951 16355 heartbeater.cc:503] Master 127.8.218.253:40141 was elected leader, sending a full tablet report...
I20260928 10:44:54.347256 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.340753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.194:53839)
0928 10:44:54.346650 (+ 5897us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.346977 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5774,"thread_start_us":5735,"threads_started":1}
I20260928 10:44:54.348726 16598 tablet_bootstrap.cc:492] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: No bootstrap required, opened a new log
I20260928 10:44:54.348898 16598 ts_tablet_manager.cc:1397] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20260928 10:44:54.349428 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.348887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.194:47229)
0928 10:44:54.349236 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.349319 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20260928 10:44:54.349711 16598 raft_consensus.cc:348] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.349902 16598 raft_consensus.cc:374] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:54.349967 16598 raft_consensus.cc:729] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251b35975dd6493f99a3a2d66bdcf20b, State: Initialized, Role: FOLLOWER
I20260928 10:44:54.350199 16598 consensus_queue.cc:260] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.350900 16598 ts_tablet_manager.cc:1428] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 10:44:54.352192 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.351277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.195:40427)
0928 10:44:54.351786 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.351930 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":277,"threads_started":1}
I20260928 10:44:54.352923 16496 heartbeater.cc:503] Master 127.8.218.253:40141 was elected leader, sending a full tablet report...
I20260928 10:44:54.353299 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.352877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.195:34605)
0928 10:44:54.353019 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.353180 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 10:44:54.519048 16356 tablet.cc:2369] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:44:54.541615 16597 raft_consensus.cc:482] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:44:54.541908 16597 raft_consensus.cc:504] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.542677 16597 leader_election.cc:288] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733), 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793)
I20260928 10:44:54.543252 16448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232ce7a99ddc4c9b85adf160dd76bcd5" candidate_uuid: "e3fd0e6225a848f88b069315eb44875a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251b35975dd6493f99a3a2d66bdcf20b" is_pre_election: true
I20260928 10:44:54.543275 16308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232ce7a99ddc4c9b85adf160dd76bcd5" candidate_uuid: "e3fd0e6225a848f88b069315eb44875a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a6c2fd3ad94007b848eb02279fd43d" is_pre_election: true
I20260928 10:44:54.543473 16448 raft_consensus.cc:2454] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fd0e6225a848f88b069315eb44875a in term 0.
I20260928 10:44:54.543514 16308 raft_consensus.cc:2454] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fd0e6225a848f88b069315eb44875a in term 0.
I20260928 10:44:54.543932 16096 leader_election.cc:302] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 251b35975dd6493f99a3a2d66bdcf20b, e3fd0e6225a848f88b069315eb44875a; no voters:
I20260928 10:44:54.544215 16597 raft_consensus.cc:2781] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:44:54.544318 16597 raft_consensus.cc:482] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:44:54.544409 16597 raft_consensus.cc:3037] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:54.546667 16597 raft_consensus.cc:504] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.547327 16597 leader_election.cc:288] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [CANDIDATE]: Term 1 election: Requested vote from peers 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733), 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793)
I20260928 10:44:54.547906 16448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232ce7a99ddc4c9b85adf160dd76bcd5" candidate_uuid: "e3fd0e6225a848f88b069315eb44875a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251b35975dd6493f99a3a2d66bdcf20b"
I20260928 10:44:54.547935 16308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232ce7a99ddc4c9b85adf160dd76bcd5" candidate_uuid: "e3fd0e6225a848f88b069315eb44875a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a6c2fd3ad94007b848eb02279fd43d"
I20260928 10:44:54.548137 16308 raft_consensus.cc:3037] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:54.548139 16448 raft_consensus.cc:3037] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:54.550382 16448 raft_consensus.cc:2454] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fd0e6225a848f88b069315eb44875a in term 1.
I20260928 10:44:54.550382 16308 raft_consensus.cc:2454] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fd0e6225a848f88b069315eb44875a in term 1.
I20260928 10:44:54.550858 16096 leader_election.cc:302] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 251b35975dd6493f99a3a2d66bdcf20b, 56a6c2fd3ad94007b848eb02279fd43d, e3fd0e6225a848f88b069315eb44875a; no voters:
I20260928 10:44:54.551154 16597 raft_consensus.cc:2781] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:54.551553 16597 raft_consensus.cc:686] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 1 LEADER]: Becoming Leader. State: Replica: e3fd0e6225a848f88b069315eb44875a, State: Running, Role: LEADER
I20260928 10:44:54.551833 16597 consensus_queue.cc:237] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } }
I20260928 10:44:54.552757 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.552264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.193:39301)
0928 10:44:54.552437 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.552559 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 10:44:54.554903 15832 catalog_manager.cc:5495] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fd0e6225a848f88b069315eb44875a (127.8.218.193). New cstate: current_term: 1 leader_uuid: "e3fd0e6225a848f88b069315eb44875a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3fd0e6225a848f88b069315eb44875a" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 } health_report { overall_health: UNKNOWN } } }
I20260928 10:44:54.618564 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.613937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.8.218.193:33805)
0928 10:44:54.614152 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.614156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.614175 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:44:54.614244 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.614247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.614322 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.614395 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:44:54.614768 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.615529 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.616944 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.617379 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.617522 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.617911 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:44:54.618023 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:44:54.618121 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.618235 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.618304 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 10:44:54.622133 16308 raft_consensus.cc:1261] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 1 FOLLOWER]: Refusing update from remote peer e3fd0e6225a848f88b069315eb44875a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:54.622210 16448 raft_consensus.cc:1261] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 1 FOLLOWER]: Refusing update from remote peer e3fd0e6225a848f88b069315eb44875a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:44:54.622916 16605 consensus_queue.cc:1035] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a6c2fd3ad94007b848eb02279fd43d" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:54.623394 16597 consensus_queue.cc:1035] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [LEADER]: Connected to new peer: Peer: permanent_uuid: "251b35975dd6493f99a3a2d66bdcf20b" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:44:54.626175 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.625491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.194:47909)
0928 10:44:54.625779 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.625915 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260928 10:44:54.627048 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.626590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.195:42441)
0928 10:44:54.626747 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.626852 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 10:44:54.633143 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.632606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.195:41381)
0928 10:44:54.632816 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.632928 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 10:44:54.634408 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.632582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.193:59299)
0928 10:44:54.633091 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.633193 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391}
I20260928 10:44:54.640096 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.639578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.193:49661)
0928 10:44:54.639777 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.639878 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 10:44:54.643467 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.642946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.8.218.194:50341)
0928 10:44:54.643176 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.643280 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 10:44:54.646258 15832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:54.646492 15832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:44:54.653010 15832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6be0af7dde2947cd98e2fae61320ea19: Sending DeleteTablet for 3 replicas of tablet 232ce7a99ddc4c9b85adf160dd76bcd5
I20260928 10:44:54.654173 16288 tablet_service.cc:1514] Processing DeleteTablet for tablet 232ce7a99ddc4c9b85adf160dd76bcd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:54 UTC) from {username='oryx', principal='oryx/127.8.218.253@KRBTEST.COM'} at 127.0.0.1:60290
I20260928 10:44:54.654259 16142 tablet_service.cc:1514] Processing DeleteTablet for tablet 232ce7a99ddc4c9b85adf160dd76bcd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:54 UTC) from {username='oryx', principal='oryx/127.8.218.253@KRBTEST.COM'} at 127.0.0.1:48468
I20260928 10:44:54.654364 16429 tablet_service.cc:1514] Processing DeleteTablet for tablet 232ce7a99ddc4c9b85adf160dd76bcd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:44:54 UTC) from {username='oryx', principal='oryx/127.8.218.253@KRBTEST.COM'} at 127.0.0.1:53886
I20260928 10:44:54.655694 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 16003
I20260928 10:44:54.666263 16619 tablet_replica.cc:330] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: stopping tablet replica
I20260928 10:44:54.666766 16619 raft_consensus.cc:2229] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:54.669428 16617 tablet_replica.cc:330] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: stopping tablet replica
I20260928 10:44:54.669675 16617 raft_consensus.cc:2229] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:44:54.669987 16617 raft_consensus.cc:2258] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:54.670079 16618 tablet_replica.cc:330] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: stopping tablet replica
I20260928 10:44:54.670285 16618 raft_consensus.cc:2229] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:44:54.670807 16618 raft_consensus.cc:2258] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:54.670976 16617 ts_tablet_manager.cc:1905] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:54.671764 16618 ts_tablet_manager.cc:1905] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:54.678292 16617 ts_tablet_manager.cc:1918] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:54.678601 16617 log.cc:1196] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/232ce7a99ddc4c9b85adf160dd76bcd5
I20260928 10:44:54.679155 16617 ts_tablet_manager.cc:1939] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 56a6c2fd3ad94007b848eb02279fd43d: Deleting consensus metadata
I20260928 10:44:54.679317 16618 ts_tablet_manager.cc:1918] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:54.679471 16618 log.cc:1196] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/232ce7a99ddc4c9b85adf160dd76bcd5
I20260928 10:44:54.679900 16618 ts_tablet_manager.cc:1939] T 232ce7a99ddc4c9b85adf160dd76bcd5 P e3fd0e6225a848f88b069315eb44875a: Deleting consensus metadata
I20260928 10:44:54.680363 15820 catalog_manager.cc:4841] TS 56a6c2fd3ad94007b848eb02279fd43d (127.8.218.194:41733): tablet 232ce7a99ddc4c9b85adf160dd76bcd5 (table test-table [id=0c7e09922564480b9db3a1a26def7adf]) successfully deleted
I20260928 10:44:54.680972 15818 catalog_manager.cc:4841] TS e3fd0e6225a848f88b069315eb44875a (127.8.218.193:33805): tablet 232ce7a99ddc4c9b85adf160dd76bcd5 (table test-table [id=0c7e09922564480b9db3a1a26def7adf]) successfully deleted
I20260928 10:44:54.681731 16619 raft_consensus.cc:2258] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:44:54.682888 16619 ts_tablet_manager.cc:1905] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:44:54.690364 16619 ts_tablet_manager.cc:1918] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:44:54.690511 16619 log.cc:1196] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/232ce7a99ddc4c9b85adf160dd76bcd5
I20260928 10:44:54.690953 16619 ts_tablet_manager.cc:1939] T 232ce7a99ddc4c9b85adf160dd76bcd5 P 251b35975dd6493f99a3a2d66bdcf20b: Deleting consensus metadata
I20260928 10:44:54.691803 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.657091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.8.218.193:33805)
0928 10:44:54.657311 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.657318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.670604 (+ 13286us) server_negotiation.cc:407] Connection header received
0928 10:44:54.670701 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.670705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.670836 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.670982 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:54.673214 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.674416 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.676414 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.677144 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.677271 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.685297 (+ 8026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:54.685316 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:54.685334 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:54.685381 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:54.688621 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:54.689522 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:54.689531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:54.689534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:54.689701 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:54.690128 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:54.690133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:54.690135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:54.690540 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:54.690800 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.691300 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.691513 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:44:54.692206 15818 catalog_manager.cc:4841] TS 251b35975dd6493f99a3a2d66bdcf20b (127.8.218.195:33793): tablet 232ce7a99ddc4c9b85adf160dd76bcd5 (table test-table [id=0c7e09922564480b9db3a1a26def7adf]) successfully deleted
I20260928 10:44:54.692517 16202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:54.820973 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 16003
I20260928 10:44:54.842746 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 16219
I20260928 10:44:54.856559 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.843573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.8.218.194:41733)
0928 10:44:54.844012 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.844017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.844033 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:44:54.844088 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.844092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.844149 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.844241 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:54.845710 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.846422 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.847742 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.848178 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.848279 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.850866 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:54.850881 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:54.850897 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:54.850935 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:54.853788 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:54.854560 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:54.854568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:54.854570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:54.854739 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:54.855205 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:54.855208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:54.855210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:54.855559 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:54.855824 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.856124 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.856304 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260928 10:44:54.857259 16348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:54.899085 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.898360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54298)
0928 10:44:54.898795 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.898920 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":223,"threads_started":1}
I20260928 10:44:54.900741 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.900336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38155 (local address 127.0.0.1:54310)
0928 10:44:54.900489 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:54.900572 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 10:44:54.952479 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 16219
I20260928 10:44:54.968107 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 16366
I20260928 10:44:54.982502 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:54.968964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.8.218.195:33793)
0928 10:44:54.969498 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:54.969545 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:54.969574 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:44:54.969636 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:54.969640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:54.969735 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:54.969845 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:54.971287 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.972022 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.973247 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:54.973785 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:54.973902 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:54.976459 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:54.976475 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:54.976491 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:54.976529 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:54.979562 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:54.980330 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:54.980338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:54.980341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:54.980474 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:54.980863 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:54.980868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:54.980870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:54.981266 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:54.981600 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:54.981932 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 10:44:54.982187 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":249,"threads_started":1}
I20260928 10:44:54.983175 16489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:55.076440 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 16366
I20260928 10:44:55.091151 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 15696
I20260928 10:44:55.092181 15863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:55.216452 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 15696
W20260928 10:44:55.234683 16016 connection.cc:535] server connection from 127.0.0.1:46504 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 10:44:55.235029 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 15797
I20260928 10:44:55.236357 16059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:44:55.354094 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 15797
2026-09-28T10:44:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9819 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:44:55 dist-test-slave-mn38 krb5kdc[16632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:44:55 dist-test-slave-mn38 krb5kdc[16632](info): set up 2 sockets
Sep 28 10:44:55 dist-test-slave-mn38 krb5kdc[16632](info): commencing operation
krb5kdc: starting...
W20260928 10:44:56.383347 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:44:56 dist-test-slave-mn38 krb5kdc[16632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:44:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.254.keytab.
Entry for principal oryx/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.254.keytab.
I20260928 10:44:56.623109 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37979
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.254.keytab
--principal=oryx/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:37979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:56.980406 16645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:56.981047 16645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:56.981191 16645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:56.988986 16645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:44:56.989120 16645 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 10:44:56.989159 16645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:56.989224 16645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:56.989274 16645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:44:56.989334 16645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:44:56.999975 16645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:37979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.254.keytab
--principal=oryx/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:57.001425 16645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:57.002898 16645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:57.015532 16650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:57.015461 16653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:57.015511 16651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:57.016415 16645 server_base.cc:1034] running on GCE node
I20260928 10:44:57.017236 16645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:57.018851 16645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:57.020089 16645 hybrid_clock.cc:648] HybridClock initialized: now 1790592297020021 us; error 48 us; skew 500 ppm
I20260928 10:44:57.020506 16645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:57 dist-test-slave-mn38 krb5kdc[16632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592297, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:57.023825 16645 init.cc:377] Logged in from keytab as oryx/127.8.218.254@KRBTEST.COM (short username oryx)
I20260928 10:44:57.025468 16645 webserver.cc:460] Webserver started at http://127.8.218.254:44223/ using document root <none> and password file <none>
I20260928 10:44:57.026180 16645 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:57.026278 16645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:57.026566 16645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:57.029820 16645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d"
format_stamp: "Formatted at 2026-09-28 10:44:57 on dist-test-slave-mn38"
server_key: "36ab83b145a77c00e04eabc2df279b0a"
server_key_iv: "45e5ac92868e219e8573444c4624e586"
server_key_version: "encryptionkey@0"
I20260928 10:44:57.030737 16645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d"
format_stamp: "Formatted at 2026-09-28 10:44:57 on dist-test-slave-mn38"
server_key: "36ab83b145a77c00e04eabc2df279b0a"
server_key_iv: "45e5ac92868e219e8573444c4624e586"
server_key_version: "encryptionkey@0"
I20260928 10:44:57.036715 16645 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:44:57.041545 16660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:57.043347 16645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:44:57.043584 16645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d"
format_stamp: "Formatted at 2026-09-28 10:44:57 on dist-test-slave-mn38"
server_key: "36ab83b145a77c00e04eabc2df279b0a"
server_key_iv: "45e5ac92868e219e8573444c4624e586"
server_key_version: "encryptionkey@0"
I20260928 10:44:57.043843 16645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:57.069240 16645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:57.074667 16645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:57.075048 16645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:57.096892 16645 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:37979
I20260928 10:44:57.096957 16721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:37979 every 8 connection(s)
I20260928 10:44:57.098443 16645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:44:57.103081 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:44:57.105964 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 16645
I20260928 10:44:57.106196 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:44:57.106952 9067 external_mini_cluster.cc:1428] Setting key 1c81a99b6f8d562aca6481e8f50db120
I20260928 10:44:57.116817 16722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: Bootstrap starting.
I20260928 10:44:57.121011 16722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: Neither blocks nor log segments found. Creating new log.
I20260928 10:44:57.122567 16722 log.cc:824] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:44:57 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:57.126780 16722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: No bootstrap required, opened a new log
I20260928 10:44:57.134438 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:57.109404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.8.218.254:37979)
0928 10:44:57.110300 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:57.110314 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:57.110362 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:44:57.112204 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:57.112232 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:57.112732 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:57.113151 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:57.115110 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.116616 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.118490 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.119269 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.119474 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:57.125318 (+ 5844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:57.125339 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:57.125360 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:57.125408 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:57.129424 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:57.130490 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:57.130499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:57.130507 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:57.130714 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:57.131202 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:57.131206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:57.131209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:57.131915 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:57.132193 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:57.132684 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 10:44:57.133142 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":371,"threads_started":1}
I20260928 10:44:57.136124 16722 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37979 } }
I20260928 10:44:57.136520 16722 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:44:57.136637 16722 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ea3c0a109a4f5bbe4386eda5389d3d, State: Initialized, Role: FOLLOWER
I20260928 10:44:57.137259 16722 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37979 } }
I20260928 10:44:57.137539 16722 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:44:57.137715 16722 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:44:57.137876 16722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:44:57.140120 16722 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37979 } }
I20260928 10:44:57.140726 16722 leader_election.cc:302] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ea3c0a109a4f5bbe4386eda5389d3d; no voters:
I20260928 10:44:57.141389 16722 leader_election.cc:288] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:44:57.141548 16727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:44:57.142181 16727 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [term 1 LEADER]: Becoming Leader. State: Replica: d5ea3c0a109a4f5bbe4386eda5389d3d, State: Running, Role: LEADER
I20260928 10:44:57.142829 16727 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37979 } }
I20260928 10:44:57.143891 16722 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:44:57.145880 16729 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ea3c0a109a4f5bbe4386eda5389d3d. Latest consensus state: current_term: 1 leader_uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37979 } } }
I20260928 10:44:57.146298 16729 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:57.146667 16728 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ea3c0a109a4f5bbe4386eda5389d3d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37979 } } }
I20260928 10:44:57.146979 16728 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d [sys.catalog]: This master's current role is: LEADER
I20260928 10:44:57.149708 16736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:44:57.155674 16736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:44:57.165103 16736 catalog_manager.cc:1347] Generated new cluster ID: d898d77deb2944e1bb9f3029fa3b691f
I20260928 10:44:57.165237 16736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:44:57.194851 16736 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:44:57.196317 16736 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:44:57.207598 16736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: Generated new TSK 0
I20260928 10:44:57.208523 16736 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab.
Entry for principal oryx/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab.
I20260928 10:44:57.269881 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37979
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab
--principal=oryx/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:57.620270 16748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:57.620572 16748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:57.620674 16748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:57.628259 16748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:57.628407 16748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:57.628495 16748 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:57.628578 16748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:44:57.639235 16748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab
--principal=oryx/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:57.640826 16748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:57.642882 16748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:57.658425 16753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:57.658367 16754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:57.658404 16756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:57.659148 16748 server_base.cc:1034] running on GCE node
I20260928 10:44:57.659842 16748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:57.660948 16748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:57.662233 16748 hybrid_clock.cc:648] HybridClock initialized: now 1790592297662154 us; error 53 us; skew 500 ppm
I20260928 10:44:57.662767 16748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:57 dist-test-slave-mn38 krb5kdc[16632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592297, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:57.666350 16748 init.cc:377] Logged in from keytab as oryx/127.8.218.193@KRBTEST.COM (short username oryx)
I20260928 10:44:57.668198 16748 webserver.cc:460] Webserver started at http://127.8.218.193:33541/ using document root <none> and password file <none>
I20260928 10:44:57.669040 16748 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:57.669164 16748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:57.669552 16748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:57.672647 16748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "991cc8c371f44a56b000abe0f81904f1"
format_stamp: "Formatted at 2026-09-28 10:44:57 on dist-test-slave-mn38"
server_key: "06ae0e345f58b4eca4a71cd3f67ed9f1"
server_key_iv: "6f81388880137fc4032f99f56d6c5cbd"
server_key_version: "encryptionkey@0"
I20260928 10:44:57.673513 16748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "991cc8c371f44a56b000abe0f81904f1"
format_stamp: "Formatted at 2026-09-28 10:44:57 on dist-test-slave-mn38"
server_key: "06ae0e345f58b4eca4a71cd3f67ed9f1"
server_key_iv: "6f81388880137fc4032f99f56d6c5cbd"
server_key_version: "encryptionkey@0"
I20260928 10:44:57.679358 16748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:44:57.683704 16763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:57.685618 16748 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:57.685842 16748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "991cc8c371f44a56b000abe0f81904f1"
format_stamp: "Formatted at 2026-09-28 10:44:57 on dist-test-slave-mn38"
server_key: "06ae0e345f58b4eca4a71cd3f67ed9f1"
server_key_iv: "6f81388880137fc4032f99f56d6c5cbd"
server_key_version: "encryptionkey@0"
I20260928 10:44:57.686142 16748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:57.700884 16748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:57.705803 16748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:57.706094 16748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:57.709637 16748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:57.709774 16748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:57.709928 16748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:57.710001 16748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:57 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592297, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:57.740020 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:57.710282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:35445 (local address 127.8.218.254:37979)
0928 10:44:57.715952 (+ 5670us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:57.715960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:57.715990 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:44:57.718102 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:57.718108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:57.718226 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:57.718331 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:57.720517 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.721638 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.724854 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.725724 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.725907 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:57.732014 (+ 6107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:57.732024 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:57.732030 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:57.732077 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:57.735476 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:57.736882 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:57.736889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:57.736894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:57.737091 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:57.737789 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:57.737796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:57.737799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:57.738197 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:57.738367 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:57.739286 (+ 919us) server_negotiation.cc:299] Negotiation successful
0928 10:44:57.739592 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5532,"thread_start_us":5421,"threads_started":1}
I20260928 10:44:57.741261 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:57.711140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37979 (local address 127.8.218.193:35445)
0928 10:44:57.714408 (+ 3268us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:57.714492 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:57.717634 (+ 3142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:57.718673 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:57.718683 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:57.719510 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:57.720252 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:57.720274 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.721910 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.721915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:57.724590 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:57.724623 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.726179 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.726195 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:57.726680 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:57.727777 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:57.727807 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:57.731619 (+ 3812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:57.735795 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:57.735807 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:57.735834 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:57.736677 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:57.737260 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:57.737273 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:57.737276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:57.737634 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:57.738374 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:57.738384 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:57.738853 (+ 469us) client_negotiation.cc:769] Sending connection context
0928 10:44:57.739116 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 10:44:57.739570 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2789,"thread_start_us":2440,"threads_started":1}
I20260928 10:44:57.762061 16748 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:42347
I20260928 10:44:57.762135 16883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:42347 every 8 connection(s)
I20260928 10:44:57.763511 16748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:44:57.765877 16884 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37979
I20260928 10:44:57.765930 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:57.749367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37979 (local address 127.8.218.193:39605)
0928 10:44:57.750179 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:57.750206 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:57.750403 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:57.750831 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:57.750836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:57.751613 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:57.752173 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:57.752191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.753745 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.753750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:57.755079 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:57.755095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.756269 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.756277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:57.756503 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:57.757341 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:57.757368 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:57.758675 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:57.762936 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:57.762941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:57.762944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:57.763381 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:57.763813 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:57.763818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:57.763821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:57.763957 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:57.764758 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:57.764764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:57.764994 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 10:44:57.765132 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 10:44:57.765484 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":406,"threads_started":1}
I20260928 10:44:57.766281 16884 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:57.766530 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:57.749278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:39605 (local address 127.8.218.254:37979)
0928 10:44:57.749531 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:57.749538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:57.750334 (+ 796us) server_negotiation.cc:407] Connection header received
0928 10:44:57.750551 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:57.750557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:57.750650 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:57.750791 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:57.752374 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.753549 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.755346 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:57.756039 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:57.756208 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:57.759011 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:57.759019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:57.759024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:57.759097 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:57.762746 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:57.763525 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:57.763530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:57.763533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:57.763699 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:57.764132 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:57.764139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:57.764142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:57.764576 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:57.765724 (+ 1148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:57.765864 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 10:44:57.766190 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:44:57.767835 16884 heartbeater.cc:511] Master 127.8.218.254:37979 requested a full tablet report, sending...
I20260928 10:44:57.770592 16677 ts_manager.cc:194] Registered new tserver with Master: 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347)
I20260928 10:44:57.772353 16677 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:35445
I20260928 10:44:57.773100 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 16748
I20260928 10:44:57.773351 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:44:57.774091 9067 external_mini_cluster.cc:1428] Setting key 2c84241e75729ec68e8d36f9dc54f3db
WARNING: no policy specified for oryx/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab.
Entry for principal oryx/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab.
I20260928 10:44:57.839962 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37979
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab
--principal=oryx/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:44:58.199572 16888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:58.199904 16888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:58.200002 16888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:58.207729 16888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:58.207866 16888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:58.207975 16888 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:58.208092 16888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:44:58.219759 16888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab
--principal=oryx/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:58.221782 16888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:58.223348 16888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:58.239406 16893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:58.239692 16894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:58.239773 16896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:58.240166 16888 server_base.cc:1034] running on GCE node
I20260928 10:44:58.241003 16888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:58.242167 16888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:58.243443 16888 hybrid_clock.cc:648] HybridClock initialized: now 1790592298243340 us; error 74 us; skew 500 ppm
I20260928 10:44:58.243860 16888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:58 dist-test-slave-mn38 krb5kdc[16632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592298, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:58.248034 16888 init.cc:377] Logged in from keytab as oryx/127.8.218.194@KRBTEST.COM (short username oryx)
I20260928 10:44:58.249903 16888 webserver.cc:460] Webserver started at http://127.8.218.194:41419/ using document root <none> and password file <none>
I20260928 10:44:58.250686 16888 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:58.250808 16888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:58.251137 16888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:58.254408 16888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "682eb2bee31f47c9b3d791873feee603"
format_stamp: "Formatted at 2026-09-28 10:44:58 on dist-test-slave-mn38"
server_key: "da2442b34be2931b082aa5ae4c794c57"
server_key_iv: "e45cea3c01b4b48a61634225392b06af"
server_key_version: "encryptionkey@0"
I20260928 10:44:58.255290 16888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "682eb2bee31f47c9b3d791873feee603"
format_stamp: "Formatted at 2026-09-28 10:44:58 on dist-test-slave-mn38"
server_key: "da2442b34be2931b082aa5ae4c794c57"
server_key_iv: "e45cea3c01b4b48a61634225392b06af"
server_key_version: "encryptionkey@0"
I20260928 10:44:58.261096 16888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 10:44:58.265411 16903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:58.267220 16888 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:44:58.267412 16888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "682eb2bee31f47c9b3d791873feee603"
format_stamp: "Formatted at 2026-09-28 10:44:58 on dist-test-slave-mn38"
server_key: "da2442b34be2931b082aa5ae4c794c57"
server_key_iv: "e45cea3c01b4b48a61634225392b06af"
server_key_version: "encryptionkey@0"
I20260928 10:44:58.267642 16888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:58.306275 16888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:58.312145 16888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:58.312465 16888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:58.316368 16888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:58.316514 16888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:58.316686 16888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:58.316746 16888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:58 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592298, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:58.347297 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:58.318435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37979 (local address 127.8.218.194:41495)
0928 10:44:58.319377 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:58.319436 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:58.322060 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:58.323237 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:58.323254 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:58.324311 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:58.325666 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:58.325715 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.327218 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.327223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:58.329633 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:58.329657 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.330968 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.330976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:58.331218 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:58.332026 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:58.332072 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:58.336270 (+ 4198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:58.341335 (+ 5065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:58.341347 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:58.341366 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:58.342442 (+ 1076us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:58.343285 (+ 843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:58.343292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:58.343296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:58.343597 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:58.344412 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:58.344425 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:58.345096 (+ 671us) client_negotiation.cc:769] Sending connection context
0928 10:44:58.345569 (+ 473us) client_negotiation.cc:240] Negotiation successful
0928 10:44:58.346075 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":378,"threads_started":1}
I20260928 10:44:58.348628 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:58.317704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:41495 (local address 127.8.218.254:37979)
0928 10:44:58.318214 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:58.318221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:58.319608 (+ 1387us) server_negotiation.cc:407] Connection header received
0928 10:44:58.322578 (+ 2970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:58.322584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:58.322694 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:58.322817 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:58.326010 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.327012 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.329888 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.330641 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.330806 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:58.336706 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:58.336716 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:58.336722 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:58.336780 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:58.341041 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:58.342799 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:58.342807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:58.342812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:58.343098 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:58.343748 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:58.343754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:58.343759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:58.344238 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:58.344454 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:58.348014 (+ 3560us) server_negotiation.cc:299] Negotiation successful
0928 10:44:58.348348 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":252,"threads_started":1}
I20260928 10:44:58.369145 16888 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:33061
I20260928 10:44:58.369158 17023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:33061 every 8 connection(s)
I20260928 10:44:58.370301 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:58.356292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37979 (local address 127.8.218.194:49087)
0928 10:44:58.356943 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:58.356959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:58.357111 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:58.357502 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:58.357510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:58.357971 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:58.358275 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:58.358285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.359295 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.359299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:58.360353 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:58.360367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.361113 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.361121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:58.361283 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:58.361947 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:58.361978 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:58.363324 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:58.367347 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:58.367352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:58.367355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:58.367767 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:58.368265 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:58.368272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:58.368276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:58.368461 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:58.369298 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:58.369304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:58.369554 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 10:44:58.369693 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 10:44:58.369970 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":372,"threads_started":1}
I20260928 10:44:58.370390 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:58.356213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:49087 (local address 127.8.218.254:37979)
0928 10:44:58.356377 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:58.356382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:58.357055 (+ 673us) server_negotiation.cc:407] Connection header received
0928 10:44:58.357272 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:58.357276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:58.357344 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:58.357427 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:58.358428 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.359140 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.360514 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.360914 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.361042 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:58.363594 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:58.363603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:58.363608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:58.363657 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:58.367161 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:58.367929 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:58.367936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:58.367941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:58.368105 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:58.368649 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:58.368654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:58.368657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:58.369136 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:58.369294 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:58.369841 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 10:44:58.370089 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:44:58.370900 16888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:44:58.373524 17024 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37979
I20260928 10:44:58.373752 17024 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:58.374047 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 16888
I20260928 10:44:58.374274 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:44:58.374665 17024 heartbeater.cc:511] Master 127.8.218.254:37979 requested a full tablet report, sending...
I20260928 10:44:58.374787 9067 external_mini_cluster.cc:1428] Setting key f00e689961c8b93122008f846653667d
I20260928 10:44:58.376091 16677 ts_manager.cc:194] Registered new tserver with Master: 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061)
I20260928 10:44:58.377040 16677 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:41495
WARNING: no policy specified for oryx/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab.
Entry for principal oryx/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab.
I20260928 10:44:58.424875 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37979
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab
--principal=oryx/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:44:58.776858 16884 heartbeater.cc:503] Master 127.8.218.254:37979 was elected leader, sending a full tablet report...
W20260928 10:44:58.806950 17028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:44:58.807271 17028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:44:58.807394 17028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:44:58.815070 17028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:44:58.815207 17028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:44:58.815330 17028 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 10:44:58.815431 17028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:44:58.825780 17028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab
--principal=oryx/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.218.254:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:44:58.827488 17028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:44:58.829156 17028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:44:58.843923 17033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:58.843770 17036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:44:58.843895 17034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:44:58.844468 17028 server_base.cc:1034] running on GCE node
I20260928 10:44:58.845299 17028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:44:58.846406 17028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:44:58.847651 17028 hybrid_clock.cc:648] HybridClock initialized: now 1790592298847575 us; error 47 us; skew 500 ppm
I20260928 10:44:58.848017 17028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:44:58 dist-test-slave-mn38 krb5kdc[16632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592298, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:44:58.851213 17028 init.cc:377] Logged in from keytab as oryx/127.8.218.195@KRBTEST.COM (short username oryx)
I20260928 10:44:58.852782 17028 webserver.cc:460] Webserver started at http://127.8.218.195:39103/ using document root <none> and password file <none>
I20260928 10:44:58.853508 17028 fs_manager.cc:359] Metadata directory not provided
I20260928 10:44:58.853614 17028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:44:58.853922 17028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:44:58.856787 17028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "4af58a4995744cc3b2f813dec4fbc6a1"
format_stamp: "Formatted at 2026-09-28 10:44:58 on dist-test-slave-mn38"
server_key: "3021eef9b39d965c5b19ad173d12e06a"
server_key_iv: "563a9aa8106ca25d2ccc71e8074bc3cf"
server_key_version: "encryptionkey@0"
I20260928 10:44:58.857621 17028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "4af58a4995744cc3b2f813dec4fbc6a1"
format_stamp: "Formatted at 2026-09-28 10:44:58 on dist-test-slave-mn38"
server_key: "3021eef9b39d965c5b19ad173d12e06a"
server_key_iv: "563a9aa8106ca25d2ccc71e8074bc3cf"
server_key_version: "encryptionkey@0"
I20260928 10:44:58.862915 17028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 10:44:58.866525 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:44:58.868016 17028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:44:58.868186 17028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "4af58a4995744cc3b2f813dec4fbc6a1"
format_stamp: "Formatted at 2026-09-28 10:44:58 on dist-test-slave-mn38"
server_key: "3021eef9b39d965c5b19ad173d12e06a"
server_key_iv: "563a9aa8106ca25d2ccc71e8074bc3cf"
server_key_version: "encryptionkey@0"
I20260928 10:44:58.868408 17028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:44:58.887735 17028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:44:58.892580 17028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:44:58.892921 17028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:44:58.895834 17028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:44:58.895990 17028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:44:58.896137 17028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:44:58.896203 17028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:44:58 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592298, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:44:58.921118 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:58.896697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:56647 (local address 127.8.218.254:37979)
0928 10:44:58.899307 (+ 2610us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:58.899313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:58.899339 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:44:58.900973 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:58.900980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:58.901071 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:58.901156 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:58.903699 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.904882 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.907275 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.907939 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.908084 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:58.913199 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:58.913208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:58.913214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:58.913269 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:58.916622 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:58.917964 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:58.917971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:58.917974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:58.918121 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:58.918769 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:58.918773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:58.918783 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:58.919223 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:58.919363 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:58.920476 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0928 10:44:58.920675 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2480,"thread_start_us":2376,"threads_started":1}
I20260928 10:44:58.923076 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:58.897512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37979 (local address 127.8.218.195:56647)
0928 10:44:58.898546 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:58.898607 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:58.900620 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:58.901755 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:58.901768 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:58.902574 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:58.903446 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:58.903481 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.905078 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.905084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:58.907103 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:58.907118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.908213 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.908223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:58.908500 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:58.909303 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:58.909338 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:58.912845 (+ 3507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:58.916848 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:58.916860 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:58.916884 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:58.917779 (+ 895us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:58.918249 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:58.918254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:58.918259 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:58.918622 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:58.919373 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:58.919384 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:58.919968 (+ 584us) client_negotiation.cc:769] Sending connection context
0928 10:44:58.920332 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 10:44:58.921054 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":382,"threads_started":1}
I20260928 10:44:58.938849 17163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:40945 every 8 connection(s)
I20260928 10:44:58.938863 17028 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:40945
I20260928 10:44:58.940899 17028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:44:58.941721 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 17028
I20260928 10:44:58.941928 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:44:58.942534 9067 external_mini_cluster.cc:1428] Setting key 1a0bc4d399b7bc767133873d1738ca40
I20260928 10:44:58.944960 17164 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37979
I20260928 10:44:58.945798 17164 heartbeater.cc:464] Registering TS with master...
I20260928 10:44:58.946414 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:58.930214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:47837 (local address 127.8.218.254:37979)
0928 10:44:58.930407 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:58.930413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:58.931402 (+ 989us) server_negotiation.cc:407] Connection header received
0928 10:44:58.931542 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:58.931548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:58.931645 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:58.931774 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:58.932760 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.934007 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.936024 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.936613 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.936701 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:58.939167 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:44:58.939175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:44:58.939180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:44:58.939225 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:44:58.943029 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:58.944274 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:58.944280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:58.944283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:58.944470 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:44:58.944937 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:44:58.944942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:44:58.944946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:44:58.945264 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:44:58.945464 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 10:44:58.945908 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 10:44:58.946151 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:44:58.946753 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:44:58.930595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37979 (local address 127.8.218.195:47837)
0928 10:44:58.931258 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 10:44:58.931283 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:44:58.931432 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:44:58.931798 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:44:58.931801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:44:58.932307 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:44:58.932607 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:58.932617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.934215 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.934221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:58.935791 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:44:58.935813 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:44:58.936990 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:44:58.936995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:44:58.937147 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:58.937891 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:44:58.937920 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:44:58.938902 (+ 982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:44:58.943313 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:58.943321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:58.943327 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:58.944050 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:58.944612 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:44:58.944618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:44:58.944622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:44:58.944827 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:44:58.945377 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:44:58.945381 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:44:58.945628 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 10:44:58.945792 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 10:44:58.946295 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":343,"threads_started":1}
I20260928 10:44:58.946998 17164 heartbeater.cc:511] Master 127.8.218.254:37979 requested a full tablet report, sending...
I20260928 10:44:58.948547 16676 ts_manager.cc:194] Registered new tserver with Master: 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:44:58.949771 16676 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:56647
I20260928 10:44:58.959203 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 10:44:58 dist-test-slave-mn38 krb5kdc[16632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 10:44:59 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:44:59 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:44:59.379945 17024 heartbeater.cc:503] Master 127.8.218.254:37979 was elected leader, sending a full tablet report...
I20260928 10:44:59.952634 17164 heartbeater.cc:503] Master 127.8.218.254:37979 was elected leader, sending a full tablet report...
W20260928 10:44:59.992976 17172 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.986s user 0.000s sys 0.014s
W20260928 10:44:59.995262 17052 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:44:59.000541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.8.218.254:37979)
0928 10:44:59.000786 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 10:44:59.000793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:44:59.000994 (+ 201us) server_negotiation.cc:407] Connection header received
0928 10:44:59.001210 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:44:59.001214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:44:59.001292 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:44:59.001383 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:44:59.002896 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:59.003631 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:59.004996 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:44:59.005493 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:44:59.005615 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:44:59.994785 (+989170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54916: BlockingRecv error: recv got EOF from 127.0.0.1:54916 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 28 10:45:00 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:45:00.019413 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.000142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.8.218.254:37979)
0928 10:45:00.000369 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.000373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.000705 (+ 332us) server_negotiation.cc:407] Connection header received
0928 10:45:00.000868 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.000872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.000953 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.001053 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:00.002455 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.003290 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.004632 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.005075 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.005189 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.013400 (+ 8211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:00.013407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:00.013411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:00.013498 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:00.016256 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:00.017129 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:00.017135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:00.017137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:00.017307 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:00.017926 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:00.017931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:00.017934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:00.018254 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:00.018476 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.018862 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.019080 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:45:00.024380 16676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:45:00.031884 16676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:45:00.063128 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.047336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.8.218.195:40945)
0928 10:45:00.048556 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.048566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.049669 (+ 1103us) server_negotiation.cc:407] Connection header received
0928 10:45:00.050043 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.050053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.050362 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.050536 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:00.052035 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.053892 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.059525 (+ 5633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.061124 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.061278 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.061555 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.062608 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.062818 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":853,"threads_started":1}
I20260928 10:45:00.063169 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.047732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33061 (local address 127.0.0.1:35114)
0928 10:45:00.049532 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.049550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.049780 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.050919 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.050924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.050951 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:00.052266 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.052279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.054507 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.054514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.059257 (+ 4743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.059274 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.061941 (+ 2667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.061951 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.062267 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.062345 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.062502 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.062746 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1655,"mutex_wait_us":397,"thread_start_us":1465,"threads_started":1}
I20260928 10:45:00.063179 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.048259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.8.218.194:33061)
0928 10:45:00.048747 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.048756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.049986 (+ 1230us) server_negotiation.cc:407] Connection header received
0928 10:45:00.050136 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.050143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.050505 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.050704 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:00.052574 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.054296 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.059453 (+ 5157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.061106 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.061282 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.061560 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.062713 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.062901 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":271,"threads_started":1}
I20260928 10:45:00.064011 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.047330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:40945 (local address 127.0.0.1:51170)
0928 10:45:00.049482 (+ 2152us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.049536 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.049836 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.050844 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.050853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.050887 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:00.051844 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.051859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.054124 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.054131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.058392 (+ 4261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.058424 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.061909 (+ 3485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.061922 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.062240 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.062335 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.063371 (+ 1036us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.063530 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":63,"thread_start_us":384,"threads_started":1}
I20260928 10:45:00.068230 17094 tablet_service.cc:1467] Processing CreateTablet for tablet d376dcf076c7461bb2ca380ebc3afe58 (DEFAULT_TABLE table=test-table [id=7e8f0771fc8a4e8b8bd4ea94b1ca266f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.068231 16954 tablet_service.cc:1467] Processing CreateTablet for tablet d376dcf076c7461bb2ca380ebc3afe58 (DEFAULT_TABLE table=test-table [id=7e8f0771fc8a4e8b8bd4ea94b1ca266f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.069782 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d376dcf076c7461bb2ca380ebc3afe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.069772 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d376dcf076c7461bb2ca380ebc3afe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.074369 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.049535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:42347 (local address 127.0.0.1:36834)
0928 10:45:00.058763 (+ 9228us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.058786 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.058911 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.063518 (+ 4607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.063524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.063558 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:00.064046 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.064063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.067596 (+ 3533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.067602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.069661 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.069685 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.071914 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.071921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.072137 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.072181 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.072284 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.073988 (+ 1704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9077,"thread_start_us":8925,"threads_started":1}
I20260928 10:45:00.075691 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.049760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.8.218.193:42347)
0928 10:45:00.062700 (+ 12940us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.062709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.062739 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:45:00.062850 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.062858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.063196 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.063525 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:00.064373 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.065838 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.069961 (+ 4123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.071505 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.074776 (+ 3271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.075066 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.075162 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.075370 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12755,"thread_start_us":12597,"threads_started":1}
I20260928 10:45:00.081676 16814 tablet_service.cc:1467] Processing CreateTablet for tablet d376dcf076c7461bb2ca380ebc3afe58 (DEFAULT_TABLE table=test-table [id=7e8f0771fc8a4e8b8bd4ea94b1ca266f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.083272 17189 tablet_bootstrap.cc:492] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Bootstrap starting.
I20260928 10:45:00.083353 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d376dcf076c7461bb2ca380ebc3afe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.085932 17190 tablet_bootstrap.cc:492] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: Bootstrap starting.
I20260928 10:45:00.087357 17189 tablet_bootstrap.cc:654] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.089165 17189 log.cc:824] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:00.090408 17190 tablet_bootstrap.cc:654] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.092216 17190 log.cc:824] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:00.094525 17189 tablet_bootstrap.cc:492] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: No bootstrap required, opened a new log
I20260928 10:45:00.094887 17189 ts_tablet_manager.cc:1397] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260928 10:45:00.096107 17190 tablet_bootstrap.cc:492] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: No bootstrap required, opened a new log
I20260928 10:45:00.096511 17190 ts_tablet_manager.cc:1397] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260928 10:45:00.106594 17189 raft_consensus.cc:348] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.108150 17193 tablet_bootstrap.cc:492] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: Bootstrap starting.
I20260928 10:45:00.108453 17189 raft_consensus.cc:374] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.108156 17190 raft_consensus.cc:348] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.108644 17189 raft_consensus.cc:729] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af58a4995744cc3b2f813dec4fbc6a1, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.108769 17190 raft_consensus.cc:374] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.108927 17190 raft_consensus.cc:729] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682eb2bee31f47c9b3d791873feee603, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.109652 17189 consensus_queue.cc:260] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.109701 17190 consensus_queue.cc:260] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.113756 17193 tablet_bootstrap.cc:654] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.115917 17193 log.cc:824] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:00.125535 17195 raft_consensus.cc:482] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:45:00.126207 17189 ts_tablet_manager.cc:1428] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent starting tablet: real 0.031s user 0.018s sys 0.012s
I20260928 10:45:00.126052 17195 raft_consensus.cc:504] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.128276 17195 leader_election.cc:288] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945), 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347)
I20260928 10:45:00.129324 17193 tablet_bootstrap.cc:492] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: No bootstrap required, opened a new log
I20260928 10:45:00.129827 17193 ts_tablet_manager.cc:1397] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20260928 10:45:00.130122 17190 ts_tablet_manager.cc:1428] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: Time spent starting tablet: real 0.033s user 0.015s sys 0.003s
I20260928 10:45:00.138705 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.129178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:38001 (local address 127.8.218.195:40945)
0928 10:45:00.129387 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.129394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.131074 (+ 1680us) server_negotiation.cc:407] Connection header received
0928 10:45:00.131361 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.131366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.131469 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.131532 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:00.132385 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.133504 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.136222 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.137567 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.137769 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.137854 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.138285 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.138447 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:45:00.138850 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.128668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:40945 (local address 127.8.218.194:38001)
0928 10:45:00.130953 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.130976 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.131194 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.131750 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.131754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.131780 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:00.132191 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.132205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.133718 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.133724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.136034 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.136050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.137846 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.137854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.138046 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.138117 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.138219 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.138375 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2140,"mutex_wait_us":1,"thread_start_us":2712,"threads_started":1}
I20260928 10:45:00.140353 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d376dcf076c7461bb2ca380ebc3afe58" candidate_uuid: "682eb2bee31f47c9b3d791873feee603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" is_pre_election: true
I20260928 10:45:00.140614 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.132534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:57313 (local address 127.8.218.193:42347)
0928 10:45:00.132762 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.132768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.133675 (+ 907us) server_negotiation.cc:407] Connection header received
0928 10:45:00.133999 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.134005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.134104 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.134168 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:00.135258 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.135990 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.138369 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.139686 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.139854 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.139972 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.140233 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.140366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:45:00.140728 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.131867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:42347 (local address 127.8.218.194:57313)
0928 10:45:00.133546 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.133567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.133802 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.134368 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.134376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.134680 (+ 304us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:00.135075 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.135085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.136123 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.136129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.138160 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.138174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.139894 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.139901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.140075 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.140100 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.140186 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.140344 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"mutex_wait_us":46,"thread_start_us":2546,"threads_started":1}
I20260928 10:45:00.140980 17114 raft_consensus.cc:2454] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 682eb2bee31f47c9b3d791873feee603 in term 0.
I20260928 10:45:00.141202 17193 raft_consensus.cc:348] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.141795 17193 raft_consensus.cc:374] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.141805 16906 leader_election.cc:302] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af58a4995744cc3b2f813dec4fbc6a1, 682eb2bee31f47c9b3d791873feee603; no voters:
I20260928 10:45:00.141919 17193 raft_consensus.cc:729] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991cc8c371f44a56b000abe0f81904f1, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.141733 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d376dcf076c7461bb2ca380ebc3afe58" candidate_uuid: "682eb2bee31f47c9b3d791873feee603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991cc8c371f44a56b000abe0f81904f1" is_pre_election: true
I20260928 10:45:00.142280 17196 raft_consensus.cc:2781] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:45:00.142432 17196 raft_consensus.cc:482] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:45:00.142557 17196 raft_consensus.cc:3037] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.142714 17193 consensus_queue.cc:260] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.143219 16835 raft_consensus.cc:2454] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 682eb2bee31f47c9b3d791873feee603 in term 0.
I20260928 10:45:00.146054 17196 raft_consensus.cc:504] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.146371 17193 ts_tablet_manager.cc:1428] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260928 10:45:00.146888 17196 leader_election.cc:288] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [CANDIDATE]: Term 1 election: Requested vote from peers 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945), 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347)
I20260928 10:45:00.147619 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d376dcf076c7461bb2ca380ebc3afe58" candidate_uuid: "682eb2bee31f47c9b3d791873feee603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1"
I20260928 10:45:00.147927 17114 raft_consensus.cc:3037] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.148705 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d376dcf076c7461bb2ca380ebc3afe58" candidate_uuid: "682eb2bee31f47c9b3d791873feee603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991cc8c371f44a56b000abe0f81904f1"
I20260928 10:45:00.148934 16835 raft_consensus.cc:3037] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.151356 17114 raft_consensus.cc:2454] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 682eb2bee31f47c9b3d791873feee603 in term 1.
I20260928 10:45:00.151983 16906 leader_election.cc:302] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af58a4995744cc3b2f813dec4fbc6a1, 682eb2bee31f47c9b3d791873feee603; no voters:
I20260928 10:45:00.152112 16835 raft_consensus.cc:2454] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 682eb2bee31f47c9b3d791873feee603 in term 1.
I20260928 10:45:00.152352 17196 raft_consensus.cc:2781] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:00.152547 17196 raft_consensus.cc:686] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 1 LEADER]: Becoming Leader. State: Replica: 682eb2bee31f47c9b3d791873feee603, State: Running, Role: LEADER
I20260928 10:45:00.153174 17196 consensus_queue.cc:237] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } }
I20260928 10:45:00.157899 16677 catalog_manager.cc:5495] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 reported cstate change: term changed from 0 to 1, leader changed from <none> to 682eb2bee31f47c9b3d791873feee603 (127.8.218.194). New cstate: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } health_report { overall_health: HEALTHY } } }
I20260928 10:45:00.192054 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.185982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.8.218.194:33061)
0928 10:45:00.186213 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.186219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.186241 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:45:00.186350 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.186354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.186450 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.186598 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:45:00.186997 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.187863 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.189294 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.189860 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.189992 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.190531 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:45:00.191288 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:45:00.191423 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.191678 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.191769 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 10:45:00.195973 17165 tablet.cc:2369] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:45:00.202419 17114 raft_consensus.cc:1261] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Refusing update from remote peer 682eb2bee31f47c9b3d791873feee603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:00.203240 16835 raft_consensus.cc:1261] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Refusing update from remote peer 682eb2bee31f47c9b3d791873feee603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:00.203861 17195 consensus_queue.cc:1035] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:00.204260 17196 consensus_queue.cc:1035] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:00.219480 17203 mvcc.cc:205] Tried to move back new op lower bound from 7334266061612769280 to 7334266061431365632. Current Snapshot: MvccSnapshot[applied={T|T < 7334266061612769280}]
I20260928 10:45:00.222122 17204 mvcc.cc:205] Tried to move back new op lower bound from 7334266061612769280 to 7334266061431365632. Current Snapshot: MvccSnapshot[applied={T|T < 7334266061612769280}]
I20260928 10:45:00.239847 16675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:00.240207 16675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:00.244940 16675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: Sending DeleteTablet for 3 replicas of tablet d376dcf076c7461bb2ca380ebc3afe58
I20260928 10:45:00.246356 16954 tablet_service.cc:1514] Processing DeleteTablet for tablet d376dcf076c7461bb2ca380ebc3afe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:00 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:35114
I20260928 10:45:00.246881 17094 tablet_service.cc:1514] Processing DeleteTablet for tablet d376dcf076c7461bb2ca380ebc3afe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:00 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:51170
I20260928 10:45:00.247720 16814 tablet_service.cc:1514] Processing DeleteTablet for tablet d376dcf076c7461bb2ca380ebc3afe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:00 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:36834
I20260928 10:45:00.247936 16677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:45:00.248711 16677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:45:00.251593 17214 tablet_replica.cc:330] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: stopping tablet replica
I20260928 10:45:00.252276 17214 raft_consensus.cc:2229] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:00.252887 17214 raft_consensus.cc:2258] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:00.258296 16813 tablet_service.cc:1467] Processing CreateTablet for tablet 046e46ff110243ed9e4263da14d1cae0 (DEFAULT_TABLE table=test-table [id=73eac37430c54678a6b2c4f8852b56ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.258831 16813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046e46ff110243ed9e4263da14d1cae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.259049 16953 tablet_service.cc:1467] Processing CreateTablet for tablet 046e46ff110243ed9e4263da14d1cae0 (DEFAULT_TABLE table=test-table [id=73eac37430c54678a6b2c4f8852b56ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.259677 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046e46ff110243ed9e4263da14d1cae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.262378 17094 tablet_service.cc:1467] Processing CreateTablet for tablet 046e46ff110243ed9e4263da14d1cae0 (DEFAULT_TABLE table=test-table [id=73eac37430c54678a6b2c4f8852b56ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.262883 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046e46ff110243ed9e4263da14d1cae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.262920 17215 tablet_replica.cc:330] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: stopping tablet replica
I20260928 10:45:00.264817 17215 raft_consensus.cc:2229] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:00.265354 17215 raft_consensus.cc:2258] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:00.266201 17213 tablet_replica.cc:330] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: stopping tablet replica
I20260928 10:45:00.266764 17213 raft_consensus.cc:2229] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:45:00.266904 17190 tablet_bootstrap.cc:492] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: Bootstrap starting.
I20260928 10:45:00.268822 17190 tablet_bootstrap.cc:654] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.269166 17214 ts_tablet_manager.cc:1905] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:00.268823 17213 raft_consensus.cc:2258] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:00.276029 17213 ts_tablet_manager.cc:1905] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:00.277070 17214 ts_tablet_manager.cc:1918] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:00.277544 17214 log.cc:1196] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/d376dcf076c7461bb2ca380ebc3afe58
I20260928 10:45:00.278237 17214 ts_tablet_manager.cc:1939] T d376dcf076c7461bb2ca380ebc3afe58 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting consensus metadata
I20260928 10:45:00.280299 17193 tablet_bootstrap.cc:492] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: Bootstrap starting.
I20260928 10:45:00.280560 16663 catalog_manager.cc:4841] TS 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945): tablet d376dcf076c7461bb2ca380ebc3afe58 (table test-table [id=7e8f0771fc8a4e8b8bd4ea94b1ca266f]) successfully deleted
I20260928 10:45:00.281069 17189 tablet_bootstrap.cc:492] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: Bootstrap starting.
I20260928 10:45:00.282234 17193 tablet_bootstrap.cc:654] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.282546 17189 tablet_bootstrap.cc:654] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.284706 17190 tablet_bootstrap.cc:492] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: No bootstrap required, opened a new log
I20260928 10:45:00.284996 17190 ts_tablet_manager.cc:1397] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260928 10:45:00.286089 17190 raft_consensus.cc:348] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.286526 17190 raft_consensus.cc:374] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.286695 17190 raft_consensus.cc:729] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682eb2bee31f47c9b3d791873feee603, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.287482 17215 ts_tablet_manager.cc:1905] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:00.287575 17190 consensus_queue.cc:260] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.291810 17190 ts_tablet_manager.cc:1428] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 10:45:00.298794 17189 tablet_bootstrap.cc:492] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: No bootstrap required, opened a new log
I20260928 10:45:00.298992 17189 ts_tablet_manager.cc:1397] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20260928 10:45:00.299257 17215 ts_tablet_manager.cc:1918] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:00.299453 17215 log.cc:1196] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/d376dcf076c7461bb2ca380ebc3afe58
I20260928 10:45:00.300010 17215 ts_tablet_manager.cc:1939] T d376dcf076c7461bb2ca380ebc3afe58 P 991cc8c371f44a56b000abe0f81904f1: Deleting consensus metadata
I20260928 10:45:00.299866 17189 raft_consensus.cc:348] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.300302 17189 raft_consensus.cc:374] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.300429 17189 raft_consensus.cc:729] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af58a4995744cc3b2f813dec4fbc6a1, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.300750 17189 consensus_queue.cc:260] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.302109 16663 catalog_manager.cc:4841] TS 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347): tablet d376dcf076c7461bb2ca380ebc3afe58 (table test-table [id=7e8f0771fc8a4e8b8bd4ea94b1ca266f]) successfully deleted
I20260928 10:45:00.302277 17189 ts_tablet_manager.cc:1428] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 10:45:00.303679 17193 tablet_bootstrap.cc:492] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: No bootstrap required, opened a new log
I20260928 10:45:00.303659 17213 ts_tablet_manager.cc:1918] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:00.303857 17193 ts_tablet_manager.cc:1397] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.004s
I20260928 10:45:00.304064 17213 log.cc:1196] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/d376dcf076c7461bb2ca380ebc3afe58
I20260928 10:45:00.304771 17213 ts_tablet_manager.cc:1939] T d376dcf076c7461bb2ca380ebc3afe58 P 682eb2bee31f47c9b3d791873feee603: Deleting consensus metadata
I20260928 10:45:00.304729 17193 raft_consensus.cc:348] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.304988 17193 raft_consensus.cc:374] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.305087 17193 raft_consensus.cc:729] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991cc8c371f44a56b000abe0f81904f1, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.305344 17193 consensus_queue.cc:260] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.306624 17193 ts_tablet_manager.cc:1428] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 10:45:00.307160 16662 catalog_manager.cc:4841] TS 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061): tablet d376dcf076c7461bb2ca380ebc3afe58 (table test-table [id=7e8f0771fc8a4e8b8bd4ea94b1ca266f]) successfully deleted
W20260928 10:45:00.374377 17025 tablet.cc:2369] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:45:00.385254 17200 raft_consensus.cc:482] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:45:00.385521 17200 raft_consensus.cc:504] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.387184 17200 leader_election.cc:288] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061), 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:45:00.406018 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.388398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:43177 (local address 127.8.218.194:33061)
0928 10:45:00.388628 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.388634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.398012 (+ 9378us) server_negotiation.cc:407] Connection header received
0928 10:45:00.398626 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.398634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.398735 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.398808 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:00.399846 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.401026 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.403271 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.404728 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.404943 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.405031 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.405543 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.405696 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:45:00.406054 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.387433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33061 (local address 127.8.218.193:43177)
0928 10:45:00.397846 (+ 10413us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.397874 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.398408 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.399068 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.399075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.399109 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:00.399645 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.399659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.401225 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.401230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.403049 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.403065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.405045 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.405052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.405270 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.405310 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.405429 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.405623 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10229,"thread_start_us":10608,"threads_started":1}
I20260928 10:45:00.407330 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046e46ff110243ed9e4263da14d1cae0" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682eb2bee31f47c9b3d791873feee603" is_pre_election: true
I20260928 10:45:00.407809 16974 raft_consensus.cc:2454] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 0.
I20260928 10:45:00.408213 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.388816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:42035 (local address 127.8.218.195:40945)
0928 10:45:00.388988 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.388993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.400716 (+ 11723us) server_negotiation.cc:407] Connection header received
0928 10:45:00.400997 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.401003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.401082 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.401146 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:00.402022 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.403249 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.405815 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.407077 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.407268 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.407347 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.407752 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.407928 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:45:00.408671 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.387760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:40945 (local address 127.8.218.193:42035)
0928 10:45:00.400598 (+ 12838us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.400621 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.400843 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.401342 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.401350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.401376 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:00.401803 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.401814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.403406 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.403411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.405601 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.405617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.407325 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.407333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.407540 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.407578 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.407760 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.408016 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12688,"thread_start_us":12959,"threads_started":1}
I20260928 10:45:00.408593 16765 leader_election.cc:302] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1; no voters:
I20260928 10:45:00.409053 17200 raft_consensus.cc:2781] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:45:00.409183 17200 raft_consensus.cc:482] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:45:00.409283 17200 raft_consensus.cc:3037] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.409631 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046e46ff110243ed9e4263da14d1cae0" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" is_pre_election: true
I20260928 10:45:00.409909 17114 raft_consensus.cc:2454] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 0.
I20260928 10:45:00.411943 17200 raft_consensus.cc:504] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.412588 17200 leader_election.cc:288] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 election: Requested vote from peers 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061), 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:45:00.413159 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046e46ff110243ed9e4263da14d1cae0" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682eb2bee31f47c9b3d791873feee603"
I20260928 10:45:00.413185 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046e46ff110243ed9e4263da14d1cae0" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1"
I20260928 10:45:00.413357 16974 raft_consensus.cc:3037] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.413415 17114 raft_consensus.cc:3037] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.415707 16974 raft_consensus.cc:2454] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 1.
I20260928 10:45:00.416208 16765 leader_election.cc:302] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1; no voters:
I20260928 10:45:00.416455 17114 raft_consensus.cc:2454] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 1.
I20260928 10:45:00.416507 17200 raft_consensus.cc:2781] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:00.417717 17200 raft_consensus.cc:686] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 1 LEADER]: Becoming Leader. State: Replica: 991cc8c371f44a56b000abe0f81904f1, State: Running, Role: LEADER
I20260928 10:45:00.418123 17200 consensus_queue.cc:237] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.422562 16675 catalog_manager.cc:5495] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193). New cstate: current_term: 1 leader_uuid: "991cc8c371f44a56b000abe0f81904f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } health_report { overall_health: UNKNOWN } } }
Sep 28 10:45:00 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592297, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.254@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:45:00.506615 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.488460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37979 (local address 127.0.0.1:54932)
0928 10:45:00.489287 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.489310 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.489513 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.489949 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.489953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.490814 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:00.491147 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.491162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.492270 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.492273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.493296 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.493309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.494560 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.494568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.494854 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.495791 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:00.495819 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:00.498782 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:00.503785 (+ 5003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:00.503800 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:00.503807 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:00.504316 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:00.504818 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:00.504822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:00.504825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:00.504999 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:00.505729 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:00.505739 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:00.505911 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.506059 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.506307 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":508,"threads_started":1}
I20260928 10:45:00.506866 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.488495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.8.218.254:37979)
0928 10:45:00.488787 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.488795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.489490 (+ 695us) server_negotiation.cc:407] Connection header received
0928 10:45:00.489648 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.489655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.489756 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.489883 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:00.491354 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.492120 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.493532 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.494260 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.494445 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.499078 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:00.499086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:00.499092 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:00.499142 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:00.503498 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:00.504473 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:00.504480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:00.504484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:00.504658 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:00.505120 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:00.505126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:00.505129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:00.505577 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:00.505725 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.506189 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.506483 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":128}
I20260928 10:45:00.513065 16675 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:54932:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 10:45:00.514269 16675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 10:45:00.517568 16885 tablet.cc:2369] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:45:00.522243 16813 tablet_service.cc:1467] Processing CreateTablet for tablet b8a7a89095ff44078ee15b7c90ce2166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50e0f8af4aaa4c038b89cb3a4aed4a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:45:00.522466 17094 tablet_service.cc:1467] Processing CreateTablet for tablet b8a7a89095ff44078ee15b7c90ce2166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50e0f8af4aaa4c038b89cb3a4aed4a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:45:00.522855 16813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a7a89095ff44078ee15b7c90ce2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.522814 16953 tablet_service.cc:1467] Processing CreateTablet for tablet b8a7a89095ff44078ee15b7c90ce2166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50e0f8af4aaa4c038b89cb3a4aed4a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 10:45:00.523041 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a7a89095ff44078ee15b7c90ce2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.523780 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a7a89095ff44078ee15b7c90ce2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.530709 17193 tablet_bootstrap.cc:492] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: Bootstrap starting.
I20260928 10:45:00.532509 17190 tablet_bootstrap.cc:492] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: Bootstrap starting.
I20260928 10:45:00.532637 17193 tablet_bootstrap.cc:654] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.534281 17189 tablet_bootstrap.cc:492] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: Bootstrap starting.
I20260928 10:45:00.534511 17190 tablet_bootstrap.cc:654] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.536139 17189 tablet_bootstrap.cc:654] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.537901 17193 tablet_bootstrap.cc:492] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: No bootstrap required, opened a new log
I20260928 10:45:00.538097 17193 ts_tablet_manager.cc:1397] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 10:45:00.538723 17190 tablet_bootstrap.cc:492] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: No bootstrap required, opened a new log
I20260928 10:45:00.538914 17190 ts_tablet_manager.cc:1397] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 10:45:00.538887 17193 raft_consensus.cc:348] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.539152 17193 raft_consensus.cc:374] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.539242 17193 raft_consensus.cc:729] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991cc8c371f44a56b000abe0f81904f1, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.539542 17193 consensus_queue.cc:260] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.539637 17190 raft_consensus.cc:348] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.539875 17190 raft_consensus.cc:374] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.539964 17190 raft_consensus.cc:729] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682eb2bee31f47c9b3d791873feee603, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.540181 17193 ts_tablet_manager.cc:1428] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 10:45:00.540035 17200 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.540343 17200 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:45:00.540246 17190 consensus_queue.cc:260] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.540680 17211 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.541020 17211 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:45:00.544653 17190 ts_tablet_manager.cc:1428] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260928 10:45:00.547072 17189 tablet_bootstrap.cc:492] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: No bootstrap required, opened a new log
I20260928 10:45:00.547278 17189 ts_tablet_manager.cc:1397] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260928 10:45:00.548069 17189 raft_consensus.cc:348] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.548327 17189 raft_consensus.cc:374] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.548409 17189 raft_consensus.cc:729] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af58a4995744cc3b2f813dec4fbc6a1, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.548691 17189 consensus_queue.cc:260] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.549381 17189 ts_tablet_manager.cc:1428] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 10:45:00.549183 17194 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.549669 17194 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:45:00.550029 17211 raft_consensus.cc:482] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:45:00.550233 17211 raft_consensus.cc:504] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.551124 17211 leader_election.cc:288] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347), 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:45:00.551741 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a7a89095ff44078ee15b7c90ce2166" candidate_uuid: "682eb2bee31f47c9b3d791873feee603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991cc8c371f44a56b000abe0f81904f1" is_pre_election: true
I20260928 10:45:00.552086 16835 raft_consensus.cc:2454] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 682eb2bee31f47c9b3d791873feee603 in term 0.
I20260928 10:45:00.552521 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a7a89095ff44078ee15b7c90ce2166" candidate_uuid: "682eb2bee31f47c9b3d791873feee603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" is_pre_election: true
I20260928 10:45:00.552794 17114 raft_consensus.cc:2454] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 682eb2bee31f47c9b3d791873feee603 in term 0.
I20260928 10:45:00.553306 16906 leader_election.cc:302] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1; no voters:
I20260928 10:45:00.553926 17211 raft_consensus.cc:2781] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:45:00.554067 17211 raft_consensus.cc:482] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:45:00.554174 17211 raft_consensus.cc:3037] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.557704 17211 raft_consensus.cc:504] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.558605 17211 leader_election.cc:288] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [CANDIDATE]: Term 1 election: Requested vote from peers 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347), 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:45:00.559510 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a7a89095ff44078ee15b7c90ce2166" candidate_uuid: "682eb2bee31f47c9b3d791873feee603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1"
I20260928 10:45:00.559664 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a7a89095ff44078ee15b7c90ce2166" candidate_uuid: "682eb2bee31f47c9b3d791873feee603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991cc8c371f44a56b000abe0f81904f1"
I20260928 10:45:00.559788 17114 raft_consensus.cc:3037] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.559893 16835 raft_consensus.cc:3037] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:00.562103 16835 raft_consensus.cc:2454] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 682eb2bee31f47c9b3d791873feee603 in term 1.
I20260928 10:45:00.562657 16906 leader_election.cc:302] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1; no voters:
I20260928 10:45:00.562811 17114 raft_consensus.cc:2454] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 682eb2bee31f47c9b3d791873feee603 in term 1.
I20260928 10:45:00.563040 17211 raft_consensus.cc:2781] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:00.563199 17211 raft_consensus.cc:686] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [term 1 LEADER]: Becoming Leader. State: Replica: 682eb2bee31f47c9b3d791873feee603, State: Running, Role: LEADER
I20260928 10:45:00.563557 17211 consensus_queue.cc:237] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.565429 17196 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: TxnStatusTablet state changed. Reason: New leader 682eb2bee31f47c9b3d791873feee603. Latest consensus state: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.565802 17196 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:45:00.566502 17233 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 10:45:00.566737 17233 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 10:45:00.567917 16677 catalog_manager.cc:5495] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 reported cstate change: term changed from 0 to 1, leader changed from <none> to 682eb2bee31f47c9b3d791873feee603 (127.8.218.194). New cstate: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } health_report { overall_health: UNKNOWN } } }
I20260928 10:45:00.592815 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.587322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33061 (local address 127.0.0.1:35142)
0928 10:45:00.587571 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.587595 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.587774 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.588198 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.588203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.588218 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:45:00.588409 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.588418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.589644 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.589649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.590771 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.590788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.591578 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.591584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.591748 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.591827 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:45:00.592252 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:45:00.592278 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.592361 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.592535 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 10:45:00.592815 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.587479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.8.218.194:33061)
0928 10:45:00.587706 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.587712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.587735 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:45:00.587990 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.587995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.588067 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.588145 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:45:00.588551 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.589421 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.590936 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.591404 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.591517 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.592017 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:45:00.592143 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:45:00.592235 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.592392 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.592483 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 10:45:00.597280 17114 raft_consensus.cc:1261] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Refusing update from remote peer 682eb2bee31f47c9b3d791873feee603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:00.597483 16835 raft_consensus.cc:1261] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Refusing update from remote peer 682eb2bee31f47c9b3d791873feee603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:00.598224 17196 consensus_queue.cc:1035] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:00.598613 17211 consensus_queue.cc:1035] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:00.602264 17200 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: TxnStatusTablet state changed. Reason: New leader 682eb2bee31f47c9b3d791873feee603. Latest consensus state: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.602632 17200 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:45:00.602700 17194 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: TxnStatusTablet state changed. Reason: New leader 682eb2bee31f47c9b3d791873feee603. Latest consensus state: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.603001 17194 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:45:00.607461 17195 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.607807 17195 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:45:00.609776 17196 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.610147 17196 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 682eb2bee31f47c9b3d791873feee603: This TxnStatusTablet replica's current role is: LEADER
I20260928 10:45:00.610463 17205 mvcc.cc:205] Tried to move back new op lower bound from 7334266063238053888 to 7334266063111135232. Current Snapshot: MvccSnapshot[applied={T|T < 7334266063238053888 or (T in {7334266063238053888})}]
I20260928 10:45:00.610916 17200 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.611215 17200 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 991cc8c371f44a56b000abe0f81904f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:45:00.613529 17194 tablet_replica.cc:439] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "682eb2bee31f47c9b3d791873feee603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } } }
I20260928 10:45:00.613796 17194 tablet_replica.cc:442] T b8a7a89095ff44078ee15b7c90ce2166 P 4af58a4995744cc3b2f813dec4fbc6a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 10:45:00.622370 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.615428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.8.218.193:42347)
0928 10:45:00.615607 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.615613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.615634 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:45:00.615743 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.615747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.615851 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.616001 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:45:00.616595 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.617695 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.619869 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.620308 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.620458 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.620972 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:45:00.621688 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:45:00.621867 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.621974 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.622068 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:45:00.640911 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.634830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:37849 (local address 127.8.218.194:33061)
0928 10:45:00.635073 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.635080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.635464 (+ 384us) server_negotiation.cc:407] Connection header received
0928 10:45:00.635666 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.635670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.635741 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.635825 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:45:00.636318 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.637129 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.638982 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.639544 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.639665 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.640138 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:45:00.640273 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:45:00.640359 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.640503 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.640587 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:45:00.640997 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.634653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33061 (local address 127.8.218.193:37849)
0928 10:45:00.635342 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.635359 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.635530 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.635850 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.635855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.635867 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:45:00.636155 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.636165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.637275 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.637281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.638802 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.638821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.639737 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.639743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.639879 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.639947 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:45:00.640376 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:45:00.640388 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.640461 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.640633 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":347,"threads_started":1}
I20260928 10:45:00.651647 17114 raft_consensus.cc:1261] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Refusing update from remote peer 991cc8c371f44a56b000abe0f81904f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:00.651928 16974 raft_consensus.cc:1261] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Refusing update from remote peer 991cc8c371f44a56b000abe0f81904f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:00.652405 17221 consensus_queue.cc:1035] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:00.652756 17200 consensus_queue.cc:1035] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:00.694773 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.687223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:43485 (local address 127.8.218.193:42347)
0928 10:45:00.687551 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.687557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.688397 (+ 840us) server_negotiation.cc:407] Connection header received
0928 10:45:00.688537 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.688542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.688638 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.688736 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:45:00.689340 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.690505 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.692605 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.693310 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.693492 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.694008 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:45:00.694155 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:45:00.694275 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.694367 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.694460 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260928 10:45:00.694914 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.687295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:42347 (local address 127.8.218.194:43485)
0928 10:45:00.688268 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:00.688292 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:00.688424 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:00.688771 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:00.688774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:00.688783 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 10:45:00.689144 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.689154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.690667 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.690673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.692432 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:00.692449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.693571 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.693576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:00.693725 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.693796 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 10:45:00.694251 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 10:45:00.694260 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 10:45:00.694346 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 10:45:00.694509 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":633,"threads_started":1}
I20260928 10:45:00.741920 16677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:00.742203 16677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:00.746490 16677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: Sending DeleteTablet for 3 replicas of tablet 046e46ff110243ed9e4263da14d1cae0
I20260928 10:45:00.747877 16813 tablet_service.cc:1514] Processing DeleteTablet for tablet 046e46ff110243ed9e4263da14d1cae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:00 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:36834
I20260928 10:45:00.747922 16953 tablet_service.cc:1514] Processing DeleteTablet for tablet 046e46ff110243ed9e4263da14d1cae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:00 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:35114
I20260928 10:45:00.748265 17213 tablet_replica.cc:330] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: stopping tablet replica
I20260928 10:45:00.748271 17215 tablet_replica.cc:330] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: stopping tablet replica
I20260928 10:45:00.748502 17213 raft_consensus.cc:2229] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:00.748519 17215 raft_consensus.cc:2229] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:45:00.748744 17213 raft_consensus.cc:2258] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:00.749112 17215 raft_consensus.cc:2258] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:00.751026 17215 ts_tablet_manager.cc:1905] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:00.754009 17094 tablet_service.cc:1514] Processing DeleteTablet for tablet 046e46ff110243ed9e4263da14d1cae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:00 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:51170
I20260928 10:45:00.754315 17214 tablet_replica.cc:330] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: stopping tablet replica
I20260928 10:45:00.754527 17214 raft_consensus.cc:2229] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:00.754738 17214 raft_consensus.cc:2258] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:00.756090 17214 ts_tablet_manager.cc:1905] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:00.766928 17215 ts_tablet_manager.cc:1918] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:45:00.769333 17214 ts_tablet_manager.cc:1918] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:45:00.769671 17214 log.cc:1196] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/046e46ff110243ed9e4263da14d1cae0
I20260928 10:45:00.770197 17214 ts_tablet_manager.cc:1939] T 046e46ff110243ed9e4263da14d1cae0 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting consensus metadata
I20260928 10:45:00.771276 17215 log.cc:1196] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/046e46ff110243ed9e4263da14d1cae0
I20260928 10:45:00.771881 16663 catalog_manager.cc:4841] TS 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945): tablet 046e46ff110243ed9e4263da14d1cae0 (table test-table [id=73eac37430c54678a6b2c4f8852b56ea]) successfully deleted
I20260928 10:45:00.772650 17215 ts_tablet_manager.cc:1939] T 046e46ff110243ed9e4263da14d1cae0 P 991cc8c371f44a56b000abe0f81904f1: Deleting consensus metadata
I20260928 10:45:00.773954 17213 ts_tablet_manager.cc:1905] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:00.774170 16663 catalog_manager.cc:4841] TS 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347): tablet 046e46ff110243ed9e4263da14d1cae0 (table test-table [id=73eac37430c54678a6b2c4f8852b56ea]) successfully deleted
I20260928 10:45:00.782060 17213 ts_tablet_manager.cc:1918] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:45:00.782280 17213 log.cc:1196] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/046e46ff110243ed9e4263da14d1cae0
I20260928 10:45:00.782747 17213 ts_tablet_manager.cc:1939] T 046e46ff110243ed9e4263da14d1cae0 P 682eb2bee31f47c9b3d791873feee603: Deleting consensus metadata
I20260928 10:45:00.784099 16662 catalog_manager.cc:4841] TS 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061): tablet 046e46ff110243ed9e4263da14d1cae0 (table test-table [id=73eac37430c54678a6b2c4f8852b56ea]) successfully deleted
Sep 28 10:45:00 dist-test-slave-mn38 krb5kdc[16632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 10:45:00 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:00 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:45:00.859715 17052 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:00.845086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.8.218.254:37979)
0928 10:45:00.845275 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.845283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.845815 (+ 532us) server_negotiation.cc:407] Connection header received
0928 10:45:00.846027 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.846033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.846122 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.846237 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:00.848038 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.849345 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.851508 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.852305 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.852463 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.859427 (+ 6964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54946: BlockingRecv error: recv got EOF from 127.0.0.1:54946 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 10:45:00 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM
I20260928 10:45:00.887641 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:00.867467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.8.218.254:37979)
0928 10:45:00.867665 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:00.867672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:00.868224 (+ 552us) server_negotiation.cc:407] Connection header received
0928 10:45:00.868403 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:00.868409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:00.868501 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:00.868609 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:00.870440 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.871624 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.873882 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:00.874553 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:00.874698 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:00.880313 (+ 5615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:00.880322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:00.880328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:00.880389 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:00.884085 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:00.885082 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:00.885089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:00.885093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:00.885268 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:00.885855 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:00.885862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:00.885865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:00.886269 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:00.886560 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:00.886944 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 10:45:00.887207 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:45:00.892347 16677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:45:00.893169 16677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:45:00.902084 16813 tablet_service.cc:1467] Processing CreateTablet for tablet 26ee840f60174ef6a56443a5c8f6ea57 (DEFAULT_TABLE table=test-table [id=f79d6bc6857c4cb2b883dd88b2e17d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.902292 16953 tablet_service.cc:1467] Processing CreateTablet for tablet 26ee840f60174ef6a56443a5c8f6ea57 (DEFAULT_TABLE table=test-table [id=f79d6bc6857c4cb2b883dd88b2e17d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.902619 16813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ee840f60174ef6a56443a5c8f6ea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.902776 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ee840f60174ef6a56443a5c8f6ea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.906345 17094 tablet_service.cc:1467] Processing CreateTablet for tablet 26ee840f60174ef6a56443a5c8f6ea57 (DEFAULT_TABLE table=test-table [id=f79d6bc6857c4cb2b883dd88b2e17d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:00.906813 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ee840f60174ef6a56443a5c8f6ea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:00.910593 17193 tablet_bootstrap.cc:492] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: Bootstrap starting.
I20260928 10:45:00.912413 17193 tablet_bootstrap.cc:654] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.923550 17193 tablet_bootstrap.cc:492] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: No bootstrap required, opened a new log
I20260928 10:45:00.923741 17193 ts_tablet_manager.cc:1397] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260928 10:45:00.924608 17193 raft_consensus.cc:348] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.924885 17193 raft_consensus.cc:374] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.924980 17193 raft_consensus.cc:729] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991cc8c371f44a56b000abe0f81904f1, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.925251 17193 consensus_queue.cc:260] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.925962 17193 ts_tablet_manager.cc:1428] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 10:45:00.927697 17189 tablet_bootstrap.cc:492] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: Bootstrap starting.
I20260928 10:45:00.927752 17190 tablet_bootstrap.cc:492] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: Bootstrap starting.
I20260928 10:45:00.929348 17189 tablet_bootstrap.cc:654] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.929418 17190 tablet_bootstrap.cc:654] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:00.946350 17190 tablet_bootstrap.cc:492] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: No bootstrap required, opened a new log
I20260928 10:45:00.946633 17190 ts_tablet_manager.cc:1397] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.009s
I20260928 10:45:00.949389 17190 raft_consensus.cc:348] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.949832 17190 raft_consensus.cc:374] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.949987 17190 raft_consensus.cc:729] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682eb2bee31f47c9b3d791873feee603, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.950351 17190 consensus_queue.cc:260] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.953783 17190 ts_tablet_manager.cc:1428] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 10:45:00.959647 17189 tablet_bootstrap.cc:492] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: No bootstrap required, opened a new log
I20260928 10:45:00.959926 17189 ts_tablet_manager.cc:1397] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.011s
I20260928 10:45:00.960896 17189 raft_consensus.cc:348] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.961330 17189 raft_consensus.cc:374] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:00.961508 17189 raft_consensus.cc:729] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af58a4995744cc3b2f813dec4fbc6a1, State: Initialized, Role: FOLLOWER
I20260928 10:45:00.961876 17189 consensus_queue.cc:260] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:00.962636 17189 ts_tablet_manager.cc:1428] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 10:45:01.171216 17221 raft_consensus.cc:482] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:45:01.171507 17221 raft_consensus.cc:504] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.172362 17221 leader_election.cc:288] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061), 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:45:01.173166 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ee840f60174ef6a56443a5c8f6ea57" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682eb2bee31f47c9b3d791873feee603" is_pre_election: true
I20260928 10:45:01.173382 17194 raft_consensus.cc:482] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:45:01.173482 16974 raft_consensus.cc:2454] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 0.
I20260928 10:45:01.173604 17194 raft_consensus.cc:504] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.174467 16765 leader_election.cc:302] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1; no voters:
I20260928 10:45:01.174937 17221 raft_consensus.cc:2781] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:45:01.175048 17221 raft_consensus.cc:482] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:45:01.175148 17221 raft_consensus.cc:3037] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:01.175395 17194 leader_election.cc:288] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347), 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061)
I20260928 10:45:01.176575 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ee840f60174ef6a56443a5c8f6ea57" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" is_pre_election: true
I20260928 10:45:01.176805 17114 raft_consensus.cc:2454] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 0.
I20260928 10:45:01.178512 17221 raft_consensus.cc:504] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.180299 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ee840f60174ef6a56443a5c8f6ea57" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682eb2bee31f47c9b3d791873feee603"
I20260928 10:45:01.180557 16974 raft_consensus.cc:3037] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:01.184389 16974 raft_consensus.cc:2454] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 1.
I20260928 10:45:01.185165 16765 leader_election.cc:302] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1; no voters:
I20260928 10:45:01.185592 17200 raft_consensus.cc:2781] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:01.185750 17200 raft_consensus.cc:686] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 1 LEADER]: Becoming Leader. State: Replica: 991cc8c371f44a56b000abe0f81904f1, State: Running, Role: LEADER
I20260928 10:45:01.186060 17200 consensus_queue.cc:237] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.189263 17221 leader_election.cc:288] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 election: Requested vote from peers 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061), 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:45:01.198688 16677 catalog_manager.cc:5495] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193). New cstate: current_term: 1 leader_uuid: "991cc8c371f44a56b000abe0f81904f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } health_report { overall_health: UNKNOWN } } }
I20260928 10:45:01.205484 17114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ee840f60174ef6a56443a5c8f6ea57" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1"
I20260928 10:45:01.205773 17114 raft_consensus.cc:3037] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:01.206939 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:01.178807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:40717 (local address 127.8.218.193:42347)
0928 10:45:01.179031 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:01.179038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:01.198739 (+ 19701us) server_negotiation.cc:407] Connection header received
0928 10:45:01.199018 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:01.199025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:01.199102 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:01.199169 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:01.199876 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.201021 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.203683 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.205281 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.205504 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:01.205602 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:01.206445 (+ 843us) server_negotiation.cc:299] Negotiation successful
0928 10:45:01.206613 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:45:01.207757 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:01.178312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:42347 (local address 127.8.218.195:40717)
0928 10:45:01.198642 (+ 20330us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:01.198661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:01.198836 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:01.199376 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:01.199381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:01.199411 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:01.199666 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:01.199678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.201240 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.201245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:01.203431 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:01.203453 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.206006 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.206013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:01.206227 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:01.206261 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:45:01.207154 (+ 893us) client_negotiation.cc:240] Negotiation successful
0928 10:45:01.207334 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20205,"thread_start_us":25779,"threads_started":1}
I20260928 10:45:01.209295 17114 raft_consensus.cc:2454] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 1.
I20260928 10:45:01.210373 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ee840f60174ef6a56443a5c8f6ea57" candidate_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991cc8c371f44a56b000abe0f81904f1" is_pre_election: true
I20260928 10:45:01.217563 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:01.176201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:41221 (local address 127.8.218.194:33061)
0928 10:45:01.211153 (+ 34952us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:01.211159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:01.211180 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:45:01.211278 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:01.211284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:01.211371 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:01.211471 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:01.212044 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.213011 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.214761 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.216035 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.216279 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:01.216369 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:01.216995 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 10:45:01.217159 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34818,"thread_start_us":34592,"threads_started":1}
I20260928 10:45:01.217584 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:01.175859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33061 (local address 127.8.218.195:41221)
0928 10:45:01.198029 (+ 22170us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:01.198057 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:01.198208 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:01.211530 (+ 13322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:01.211536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:01.211570 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:01.211867 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:01.211877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.213181 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.213187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:01.214576 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:01.214595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.216333 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.216340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:01.216682 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:01.216805 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 10:45:01.216972 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 10:45:01.217141 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22007,"thread_start_us":28778,"threads_started":1}
I20260928 10:45:01.218492 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ee840f60174ef6a56443a5c8f6ea57" candidate_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682eb2bee31f47c9b3d791873feee603" is_pre_election: true
I20260928 10:45:01.218842 16974 raft_consensus.cc:2379] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4af58a4995744cc3b2f813dec4fbc6a1 in current term 1: Already voted for candidate 991cc8c371f44a56b000abe0f81904f1 in this term.
I20260928 10:45:01.219375 17045 leader_election.cc:302] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4af58a4995744cc3b2f813dec4fbc6a1; no voters: 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1
I20260928 10:45:01.219956 17194 raft_consensus.cc:2726] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 10:45:01.233464 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:01.228208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.8.218.193:42347)
0928 10:45:01.228543 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:01.228549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:01.228605 (+ 56us) server_negotiation.cc:407] Connection header received
0928 10:45:01.228860 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:01.228865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:01.228953 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:01.229083 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:45:01.229586 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.230322 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.231677 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.232128 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.232246 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:01.232686 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:45:01.232806 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:45:01.232884 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:01.233050 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 10:45:01.233139 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260928 10:45:01.237164 17114 raft_consensus.cc:1261] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Refusing update from remote peer 991cc8c371f44a56b000abe0f81904f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:01.237164 16974 raft_consensus.cc:1261] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Refusing update from remote peer 991cc8c371f44a56b000abe0f81904f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:01.238071 17221 consensus_queue.cc:1035] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:01.238909 17221 consensus_queue.cc:1035] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:01.284771 16675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:01.285043 16675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:01.288565 16675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: Sending DeleteTablet for 3 replicas of tablet 26ee840f60174ef6a56443a5c8f6ea57
I20260928 10:45:01.289682 16813 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ee840f60174ef6a56443a5c8f6ea57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:01 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:36834
I20260928 10:45:01.290176 16953 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ee840f60174ef6a56443a5c8f6ea57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:01 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:35114
I20260928 10:45:01.290516 17213 tablet_replica.cc:330] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: stopping tablet replica
I20260928 10:45:01.290722 17213 raft_consensus.cc:2229] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:01.290794 17094 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ee840f60174ef6a56443a5c8f6ea57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:01 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:51170
I20260928 10:45:01.291004 17213 raft_consensus.cc:2258] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:01.292773 17213 ts_tablet_manager.cc:1905] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:01.291360 16675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:45:01.298868 16675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:45:01.311447 16953 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1273cd59934d82bea93afcfa2f5073 (DEFAULT_TABLE table=test-table [id=91bef27c13804d71a920fbb9da61b9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:01.312091 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1273cd59934d82bea93afcfa2f5073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:01.316533 16814 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1273cd59934d82bea93afcfa2f5073 (DEFAULT_TABLE table=test-table [id=91bef27c13804d71a920fbb9da61b9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:01.316902 17093 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1273cd59934d82bea93afcfa2f5073 (DEFAULT_TABLE table=test-table [id=91bef27c13804d71a920fbb9da61b9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:01.317330 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1273cd59934d82bea93afcfa2f5073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:01.317513 17093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1273cd59934d82bea93afcfa2f5073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:01.322304 17190 tablet_bootstrap.cc:492] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: Bootstrap starting.
I20260928 10:45:01.324506 17213 ts_tablet_manager.cc:1918] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:01.324676 17213 log.cc:1196] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/26ee840f60174ef6a56443a5c8f6ea57
I20260928 10:45:01.324781 17190 tablet_bootstrap.cc:654] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:01.325279 17189 tablet_bootstrap.cc:492] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: Bootstrap starting.
I20260928 10:45:01.327235 17189 tablet_bootstrap.cc:654] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:01.331576 17213 ts_tablet_manager.cc:1939] T 26ee840f60174ef6a56443a5c8f6ea57 P 682eb2bee31f47c9b3d791873feee603: Deleting consensus metadata
I20260928 10:45:01.333356 17276 tablet_replica.cc:330] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: stopping tablet replica
I20260928 10:45:01.333770 17193 tablet_bootstrap.cc:492] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: Bootstrap starting.
I20260928 10:45:01.334498 17276 raft_consensus.cc:2229] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:01.334820 17276 raft_consensus.cc:2258] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:01.335101 16662 catalog_manager.cc:4841] TS 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061): tablet 26ee840f60174ef6a56443a5c8f6ea57 (table test-table [id=f79d6bc6857c4cb2b883dd88b2e17d0f]) successfully deleted
I20260928 10:45:01.335656 17193 tablet_bootstrap.cc:654] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:01.337369 17276 ts_tablet_manager.cc:1905] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:01.338153 17275 tablet_replica.cc:330] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: stopping tablet replica
I20260928 10:45:01.339054 17275 raft_consensus.cc:2229] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:45:01.340559 17189 tablet_bootstrap.cc:492] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: No bootstrap required, opened a new log
I20260928 10:45:01.341193 17275 raft_consensus.cc:2258] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:01.341391 17189 ts_tablet_manager.cc:1397] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20260928 10:45:01.342752 17189 raft_consensus.cc:348] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.343312 17189 raft_consensus.cc:374] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:01.343609 17189 raft_consensus.cc:729] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af58a4995744cc3b2f813dec4fbc6a1, State: Initialized, Role: FOLLOWER
I20260928 10:45:01.344650 17189 consensus_queue.cc:260] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.346846 17190 tablet_bootstrap.cc:492] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: No bootstrap required, opened a new log
I20260928 10:45:01.347090 17190 ts_tablet_manager.cc:1397] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.005s
I20260928 10:45:01.347378 17193 tablet_bootstrap.cc:492] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: No bootstrap required, opened a new log
I20260928 10:45:01.347707 17193 ts_tablet_manager.cc:1397] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260928 10:45:01.348528 17190 raft_consensus.cc:348] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.348752 17193 raft_consensus.cc:348] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.349068 17193 raft_consensus.cc:374] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:01.349084 17190 raft_consensus.cc:374] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:01.349189 17193 raft_consensus.cc:729] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991cc8c371f44a56b000abe0f81904f1, State: Initialized, Role: FOLLOWER
I20260928 10:45:01.349248 17190 raft_consensus.cc:729] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682eb2bee31f47c9b3d791873feee603, State: Initialized, Role: FOLLOWER
I20260928 10:45:01.349601 17189 ts_tablet_manager.cc:1428] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260928 10:45:01.349567 17193 consensus_queue.cc:260] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.350805 17193 ts_tablet_manager.cc:1428] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 10:45:01.352126 17275 ts_tablet_manager.cc:1905] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:01.353971 17190 consensus_queue.cc:260] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.354624 17190 ts_tablet_manager.cc:1428] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 10:45:01.360630 17276 ts_tablet_manager.cc:1918] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:01.360924 17276 log.cc:1196] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/26ee840f60174ef6a56443a5c8f6ea57
I20260928 10:45:01.361413 17276 ts_tablet_manager.cc:1939] T 26ee840f60174ef6a56443a5c8f6ea57 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting consensus metadata
I20260928 10:45:01.362474 16663 catalog_manager.cc:4841] TS 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945): tablet 26ee840f60174ef6a56443a5c8f6ea57 (table test-table [id=f79d6bc6857c4cb2b883dd88b2e17d0f]) successfully deleted
I20260928 10:45:01.362609 17275 ts_tablet_manager.cc:1918] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:01.362762 17275 log.cc:1196] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/26ee840f60174ef6a56443a5c8f6ea57
I20260928 10:45:01.363211 17275 ts_tablet_manager.cc:1939] T 26ee840f60174ef6a56443a5c8f6ea57 P 991cc8c371f44a56b000abe0f81904f1: Deleting consensus metadata
I20260928 10:45:01.364331 16663 catalog_manager.cc:4841] TS 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347): tablet 26ee840f60174ef6a56443a5c8f6ea57 (table test-table [id=f79d6bc6857c4cb2b883dd88b2e17d0f]) successfully deleted
I20260928 10:45:01.391636 17221 raft_consensus.cc:482] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:45:01.391881 17221 raft_consensus.cc:504] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.392720 17221 leader_election.cc:288] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061), 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:45:01.393419 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1273cd59934d82bea93afcfa2f5073" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682eb2bee31f47c9b3d791873feee603" is_pre_election: true
I20260928 10:45:01.393385 17113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1273cd59934d82bea93afcfa2f5073" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" is_pre_election: true
I20260928 10:45:01.393733 16974 raft_consensus.cc:2454] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 0.
I20260928 10:45:01.393723 17113 raft_consensus.cc:2454] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 0.
I20260928 10:45:01.394412 16765 leader_election.cc:302] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1; no voters:
I20260928 10:45:01.394794 17221 raft_consensus.cc:2781] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:45:01.394903 17221 raft_consensus.cc:482] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:45:01.395035 17221 raft_consensus.cc:3037] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:01.397241 17221 raft_consensus.cc:504] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.398015 17221 leader_election.cc:288] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 election: Requested vote from peers 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061), 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945)
I20260928 10:45:01.398627 17113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1273cd59934d82bea93afcfa2f5073" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af58a4995744cc3b2f813dec4fbc6a1"
I20260928 10:45:01.398633 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1273cd59934d82bea93afcfa2f5073" candidate_uuid: "991cc8c371f44a56b000abe0f81904f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682eb2bee31f47c9b3d791873feee603"
I20260928 10:45:01.398864 16974 raft_consensus.cc:3037] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:01.398865 17113 raft_consensus.cc:3037] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:01.402137 17113 raft_consensus.cc:2454] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 1.
I20260928 10:45:01.402140 16974 raft_consensus.cc:2454] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 991cc8c371f44a56b000abe0f81904f1 in term 1.
I20260928 10:45:01.402829 16765 leader_election.cc:302] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4af58a4995744cc3b2f813dec4fbc6a1, 682eb2bee31f47c9b3d791873feee603, 991cc8c371f44a56b000abe0f81904f1; no voters:
I20260928 10:45:01.403306 17221 raft_consensus.cc:2781] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:01.403489 17221 raft_consensus.cc:686] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 1 LEADER]: Becoming Leader. State: Replica: 991cc8c371f44a56b000abe0f81904f1, State: Running, Role: LEADER
I20260928 10:45:01.403766 17221 consensus_queue.cc:237] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } }
I20260928 10:45:01.406375 16677 catalog_manager.cc:5495] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193). New cstate: current_term: 1 leader_uuid: "991cc8c371f44a56b000abe0f81904f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "991cc8c371f44a56b000abe0f81904f1" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 42347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 } health_report { overall_health: UNKNOWN } } }
I20260928 10:45:01.432085 17113 raft_consensus.cc:1261] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Refusing update from remote peer 991cc8c371f44a56b000abe0f81904f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:01.432065 16974 raft_consensus.cc:1261] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Refusing update from remote peer 991cc8c371f44a56b000abe0f81904f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:01.432739 17221 consensus_queue.cc:1035] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "682eb2bee31f47c9b3d791873feee603" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:01.433008 17200 consensus_queue.cc:1035] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af58a4995744cc3b2f813dec4fbc6a1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:01.483429 16676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:01.483753 16676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:01.486846 16676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5ea3c0a109a4f5bbe4386eda5389d3d: Sending DeleteTablet for 3 replicas of tablet 4e1273cd59934d82bea93afcfa2f5073
I20260928 10:45:01.488286 17094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1273cd59934d82bea93afcfa2f5073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:01 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:51170
I20260928 10:45:01.488255 16813 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1273cd59934d82bea93afcfa2f5073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:01 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:36834
I20260928 10:45:01.488722 17276 tablet_replica.cc:330] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: stopping tablet replica
I20260928 10:45:01.488751 16953 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1273cd59934d82bea93afcfa2f5073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:01 UTC) from {username='oryx', principal='oryx/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:35114
I20260928 10:45:01.488940 17276 raft_consensus.cc:2229] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:01.489012 17213 tablet_replica.cc:330] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: stopping tablet replica
I20260928 10:45:01.489013 17275 tablet_replica.cc:330] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: stopping tablet replica
I20260928 10:45:01.489194 17276 raft_consensus.cc:2258] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:01.489203 17213 raft_consensus.cc:2229] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:01.489203 17275 raft_consensus.cc:2229] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:45:01.489379 17213 raft_consensus.cc:2258] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:01.489570 17275 raft_consensus.cc:2258] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:01.490337 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 16748
I20260928 10:45:01.490388 17276 ts_tablet_manager.cc:1905] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:01.490603 17213 ts_tablet_manager.cc:1905] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:01.491896 17275 ts_tablet_manager.cc:1905] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:01.497741 17213 ts_tablet_manager.cc:1918] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:45:01.498610 17213 log.cc:1196] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/4e1273cd59934d82bea93afcfa2f5073
I20260928 10:45:01.499198 17213 ts_tablet_manager.cc:1939] T 4e1273cd59934d82bea93afcfa2f5073 P 682eb2bee31f47c9b3d791873feee603: Deleting consensus metadata
I20260928 10:45:01.499271 17275 ts_tablet_manager.cc:1918] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:45:01.499460 17275 log.cc:1196] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/4e1273cd59934d82bea93afcfa2f5073
I20260928 10:45:01.499917 17275 ts_tablet_manager.cc:1939] T 4e1273cd59934d82bea93afcfa2f5073 P 991cc8c371f44a56b000abe0f81904f1: Deleting consensus metadata
I20260928 10:45:01.500221 17276 ts_tablet_manager.cc:1918] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 10:45:01.500679 17276 log.cc:1196] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/4e1273cd59934d82bea93afcfa2f5073
I20260928 10:45:01.500689 16662 catalog_manager.cc:4841] TS 682eb2bee31f47c9b3d791873feee603 (127.8.218.194:33061): tablet 4e1273cd59934d82bea93afcfa2f5073 (table test-table [id=91bef27c13804d71a920fbb9da61b9e8]) successfully deleted
I20260928 10:45:01.501319 17276 ts_tablet_manager.cc:1939] T 4e1273cd59934d82bea93afcfa2f5073 P 4af58a4995744cc3b2f813dec4fbc6a1: Deleting consensus metadata
I20260928 10:45:01.502204 16663 catalog_manager.cc:4841] TS 991cc8c371f44a56b000abe0f81904f1 (127.8.218.193:42347): tablet 4e1273cd59934d82bea93afcfa2f5073 (table test-table [id=91bef27c13804d71a920fbb9da61b9e8]) successfully deleted
I20260928 10:45:01.503013 16663 catalog_manager.cc:4841] TS 4af58a4995744cc3b2f813dec4fbc6a1 (127.8.218.195:40945): tablet 4e1273cd59934d82bea93afcfa2f5073 (table test-table [id=91bef27c13804d71a920fbb9da61b9e8]) successfully deleted
Sep 28 10:45:01 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.193@KRBTEST.COM
I20260928 10:45:01.519714 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:01.493165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.8.218.193:42347)
0928 10:45:01.493388 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:01.493395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:01.502222 (+ 8827us) server_negotiation.cc:407] Connection header received
0928 10:45:01.502716 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:01.502725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:01.502852 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:01.503036 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:01.504762 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.506140 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.507661 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.508131 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.508255 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:01.512874 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:01.512892 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:01.512910 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:01.512959 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:01.516190 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:01.517206 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:01.517215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:01.517217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:01.517377 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:01.517949 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:01.517953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:01.517955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:01.518409 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:01.518948 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:01.519146 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 10:45:01.519379 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:45:01.520445 16879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:01.645540 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 16748
I20260928 10:45:01.664966 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 16888
Sep 28 10:45:01 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.194@KRBTEST.COM
I20260928 10:45:01.679566 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:01.665801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.8.218.194:33061)
0928 10:45:01.665947 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:01.665952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:01.665969 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:45:01.666113 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:01.666118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:01.666177 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:01.666259 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:01.667389 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.668112 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.669341 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.669963 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.670058 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:01.674231 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:01.674250 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:01.674267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:01.674309 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:01.677095 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:01.677865 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:01.677874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:01.677877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:01.678032 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:01.678342 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:01.678345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:01.678347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:01.678694 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:01.678913 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:01.679132 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 10:45:01.679313 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 10:45:01.680310 17019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:01.824600 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 16888
I20260928 10:45:01.846525 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 17028
Sep 28 10:45:01 dist-test-slave-mn38 krb5kdc[16632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.218.195@KRBTEST.COM
I20260928 10:45:01.862366 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:01.847447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.8.218.195:40945)
0928 10:45:01.847873 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:01.847877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:01.847894 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:45:01.847948 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:01.847952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:01.848014 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:01.848110 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:01.849411 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.850195 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.851695 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:01.852096 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:01.852217 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:01.856557 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:01.856573 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:01.856590 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:01.856658 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:01.859574 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:01.860339 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:01.860348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:01.860350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:01.860511 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:01.860869 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:01.860873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:01.860875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:01.861234 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:01.861574 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:01.861897 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 10:45:01.862106 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":244,"threads_started":1}
I20260928 10:45:01.863021 17159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:01.960160 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 17028
I20260928 10:45:01.974973 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 16645
I20260928 10:45:01.976235 16717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:02.092335 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 16645
2026-09-28T10:45:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6738 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:45:02 dist-test-slave-mn38 krb5kdc[17296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:45:02 dist-test-slave-mn38 krb5kdc[17296](info): set up 2 sockets
Sep 28 10:45:02 dist-test-slave-mn38 krb5kdc[17296](info): commencing operation
krb5kdc: starting...
W20260928 10:45:03.119282 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:45:03 dist-test-slave-mn38 krb5kdc[17296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:45:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:45:03.354415 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36803
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:36803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:03.709637 17309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:03.709978 17309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:03.710103 17309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:03.717937 17309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:45:03.718078 17309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:03.718202 17309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:03.718371 17309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:45:03.718453 17309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:45:03.730796 17309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:36803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:03.732239 17309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:03.733781 17309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:03.748736 17314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:03.748709 17315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:03.748662 17317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:03.749785 17309 server_base.cc:1034] running on GCE node
I20260928 10:45:03.750701 17309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:03.752349 17309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:03.753599 17309 hybrid_clock.cc:648] HybridClock initialized: now 1790592303753512 us; error 63 us; skew 500 ppm
I20260928 10:45:03.754048 17309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T10:45:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1734 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:45:03 dist-test-slave-mn38 krb5kdc[17323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:45:03 dist-test-slave-mn38 krb5kdc[17323](info): set up 2 sockets
Sep 28 10:45:03 dist-test-slave-mn38 krb5kdc[17323](info): commencing operation
krb5kdc: starting...
W20260928 10:45:04.862907 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:45:05 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:45:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:45:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:45:05.097271 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:42791
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:05.451686 17336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:05.451996 17336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:05.452121 17336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:05.460381 17336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:45:05.460525 17336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:05.460579 17336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:05.460661 17336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:45:05.460714 17336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:45:05.471235 17336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:42791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:05.472667 17336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:05.474085 17336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:05.486999 17342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:05.487067 17341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:05.487069 17344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:05.487841 17336 server_base.cc:1034] running on GCE node
I20260928 10:45:05.488672 17336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:05.490388 17336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:05.491652 17336 hybrid_clock.cc:648] HybridClock initialized: now 1790592305491585 us; error 53 us; skew 500 ppm
I20260928 10:45:05.492080 17336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:05 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592305, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:05.500376 17336 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:45:05.502377 17336 webserver.cc:460] Webserver started at http://127.8.218.254:41657/ using document root <none> and password file <none>
I20260928 10:45:05.503309 17336 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:05.503453 17336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:05.503793 17336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:05.507889 17336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "c00e1edf3fd640208aa08007fc26afd0"
format_stamp: "Formatted at 2026-09-28 10:45:05 on dist-test-slave-mn38"
server_key: "7af08941ca74b652932c721e3487d4f6"
server_key_iv: "2e000bb8a206682629aa9738a7753ade"
server_key_version: "encryptionkey@0"
I20260928 10:45:05.508864 17336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "c00e1edf3fd640208aa08007fc26afd0"
format_stamp: "Formatted at 2026-09-28 10:45:05 on dist-test-slave-mn38"
server_key: "7af08941ca74b652932c721e3487d4f6"
server_key_iv: "2e000bb8a206682629aa9738a7753ade"
server_key_version: "encryptionkey@0"
I20260928 10:45:05.516197 17336 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 10:45:05.520853 17351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:05.522794 17336 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:45:05.523012 17336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "c00e1edf3fd640208aa08007fc26afd0"
format_stamp: "Formatted at 2026-09-28 10:45:05 on dist-test-slave-mn38"
server_key: "7af08941ca74b652932c721e3487d4f6"
server_key_iv: "2e000bb8a206682629aa9738a7753ade"
server_key_version: "encryptionkey@0"
I20260928 10:45:05.523241 17336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:05.542685 17336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:05.547665 17336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:05.547988 17336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:05.566320 17336 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:42791
I20260928 10:45:05.566334 17402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:42791 every 8 connection(s)
I20260928 10:45:05.568065 17336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:45:05.571161 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 17336
I20260928 10:45:05.571372 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:45:05.571980 9067 external_mini_cluster.cc:1428] Setting key 50daa36be05e9c78b90658341eadfedc
I20260928 10:45:05.572536 17403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:05.587990 17403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Bootstrap starting.
Sep 28 10:45:05 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:45:05.592294 17403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:05.594063 17403 log.cc:824] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:05.598416 17403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: No bootstrap required, opened a new log
I20260928 10:45:05.600533 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:05.574137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.8.218.254:42791)
0928 10:45:05.574972 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:05.574988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:05.575039 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:45:05.577523 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:05.577556 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:05.578171 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:05.578688 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:05.580105 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:05.582205 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:05.583799 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:05.585104 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:05.585358 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:05.589891 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:05.589908 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:05.589934 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:05.590004 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:05.594555 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:05.595962 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:05.595981 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:05.595992 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:05.596269 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:05.596936 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:05.596941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:05.596945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:05.597913 (+ 968us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:05.598190 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:05.598879 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 10:45:05.599377 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":352,"threads_started":1}
I20260928 10:45:05.607820 17403 raft_consensus.cc:348] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } }
I20260928 10:45:05.608287 17403 raft_consensus.cc:374] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:05.608392 17403 raft_consensus.cc:729] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00e1edf3fd640208aa08007fc26afd0, State: Initialized, Role: FOLLOWER
I20260928 10:45:05.609113 17403 consensus_queue.cc:260] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } }
I20260928 10:45:05.609454 17403 raft_consensus.cc:388] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:45:05.609638 17403 raft_consensus.cc:482] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:45:05.609803 17403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:05.612282 17403 raft_consensus.cc:504] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } }
I20260928 10:45:05.612866 17403 leader_election.cc:302] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c00e1edf3fd640208aa08007fc26afd0; no voters:
I20260928 10:45:05.613677 17403 leader_election.cc:288] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:45:05.613818 17408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:05.614506 17408 raft_consensus.cc:686] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 1 LEADER]: Becoming Leader. State: Replica: c00e1edf3fd640208aa08007fc26afd0, State: Running, Role: LEADER
I20260928 10:45:05.615121 17408 consensus_queue.cc:237] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } }
I20260928 10:45:05.615965 17403 sys_catalog.cc:564] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:45:05.618481 17409 sys_catalog.cc:455] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c00e1edf3fd640208aa08007fc26afd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } } }
I20260928 10:45:05.618449 17410 sys_catalog.cc:455] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c00e1edf3fd640208aa08007fc26afd0. Latest consensus state: current_term: 1 leader_uuid: "c00e1edf3fd640208aa08007fc26afd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } } }
I20260928 10:45:05.619184 17409 sys_catalog.cc:458] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:05.619251 17410 sys_catalog.cc:458] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:05.620146 17415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:45:05.626739 17415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:45:05.637526 17415 catalog_manager.cc:1347] Generated new cluster ID: c22d800c95ca45f1887fad639afba443
I20260928 10:45:05.637669 17415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:45:05.666780 17415 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:45:05.668767 17415 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:45:05.680469 17415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Generated new TSK 0
I20260928 10:45:05.681708 17415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:45:05.741870 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42791
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:06.093717 17429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:06.094070 17429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:06.094202 17429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:06.102053 17429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:06.102197 17429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:06.102355 17429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:45:06.112954 17429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:06.114466 17429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:06.116127 17429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:06.130563 17434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:06.130546 17437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:06.130712 17435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:06.131543 17429 server_base.cc:1034] running on GCE node
I20260928 10:45:06.132216 17429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:06.133247 17429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:06.134451 17429 hybrid_clock.cc:648] HybridClock initialized: now 1790592306134404 us; error 34 us; skew 500 ppm
I20260928 10:45:06.134804 17429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:06 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592306, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:06.138163 17429 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:45:06.139768 17429 webserver.cc:460] Webserver started at http://127.8.218.193:33511/ using document root <none> and password file <none>
I20260928 10:45:06.140480 17429 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:06.140610 17429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:06.140902 17429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:06.143909 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "329eb79450574aa8bba9123101db61fc"
format_stamp: "Formatted at 2026-09-28 10:45:06 on dist-test-slave-mn38"
server_key: "2899058ed6a29fc017ae0c6750732ffa"
server_key_iv: "444c1da58ad9a521eae633918143b7fc"
server_key_version: "encryptionkey@0"
I20260928 10:45:06.144707 17429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "329eb79450574aa8bba9123101db61fc"
format_stamp: "Formatted at 2026-09-28 10:45:06 on dist-test-slave-mn38"
server_key: "2899058ed6a29fc017ae0c6750732ffa"
server_key_iv: "444c1da58ad9a521eae633918143b7fc"
server_key_version: "encryptionkey@0"
I20260928 10:45:06.150329 17429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 10:45:06.154398 17444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:06.155968 17429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:45:06.156178 17429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "329eb79450574aa8bba9123101db61fc"
format_stamp: "Formatted at 2026-09-28 10:45:06 on dist-test-slave-mn38"
server_key: "2899058ed6a29fc017ae0c6750732ffa"
server_key_iv: "444c1da58ad9a521eae633918143b7fc"
server_key_version: "encryptionkey@0"
I20260928 10:45:06.156388 17429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:06.174839 17429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:06.179363 17429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:06.179664 17429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:06.180737 17429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:06.182318 17429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:06.182402 17429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:06.182499 17429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:06.182542 17429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:06.214705 17429 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:41565
I20260928 10:45:06.214787 17556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:41565 every 8 connection(s)
I20260928 10:45:06.216084 17429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:45:06.224850 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 17429
I20260928 10:45:06.225082 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:45:06.225783 9067 external_mini_cluster.cc:1428] Setting key 02b32fa4fc88b5ea3d84264d7a5905d0
Sep 28 10:45:06 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592306, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:45:06.240675 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:06.219751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:56479 (local address 127.8.218.254:42791)
0928 10:45:06.220405 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:06.220410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:06.221623 (+ 1213us) server_negotiation.cc:407] Connection header received
0928 10:45:06.224048 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:06.224055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:06.224163 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:06.224262 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:06.226363 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:06.227235 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:06.229151 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:06.229818 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:06.229959 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:06.233574 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:06.233581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:06.233586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:06.233628 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:06.236847 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:06.237998 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:06.238003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:06.238005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:06.238165 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:06.238746 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:06.238751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:06.238753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:06.239055 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:06.239208 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:06.240180 (+ 972us) server_negotiation.cc:299] Negotiation successful
0928 10:45:06.240405 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":416,"threads_started":1}
I20260928 10:45:06.242091 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:06.220387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:56479)
0928 10:45:06.221337 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:06.221398 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:06.223652 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:06.224637 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:06.224651 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:06.225365 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:06.226138 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:06.226162 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:06.227451 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:06.227455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:06.228949 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:06.228966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:06.230057 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:06.230063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:06.230302 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:06.231036 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:06.231070 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:06.233278 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:06.237079 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:06.237092 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:06.237117 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:06.237805 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:06.238285 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:06.238289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:06.238292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:06.238583 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:06.239196 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:06.239205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:06.239698 (+ 493us) client_negotiation.cc:769] Sending connection context
0928 10:45:06.240029 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 10:45:06.240607 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":350,"threads_started":1}
I20260928 10:45:06.243978 17557 heartbeater.cc:347] Connected to a master server at 127.8.218.254:42791
I20260928 10:45:06.244315 17557 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:06.245193 17557 heartbeater.cc:511] Master 127.8.218.254:42791 requested a full tablet report, sending...
I20260928 10:45:06.248090 17368 ts_manager.cc:194] Registered new tserver with Master: 329eb79450574aa8bba9123101db61fc (127.8.218.193:41565)
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:45:06.249992 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:56479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:45:06.281890 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42791
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:06.637260 17563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:06.637833 17563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:06.637979 17563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:06.646193 17563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:06.646314 17563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:06.646481 17563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:45:06.656665 17563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:06.658089 17563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:06.659415 17563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:06.673616 17568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:06.673591 17569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:06.674023 17571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:06.674597 17563 server_base.cc:1034] running on GCE node
I20260928 10:45:06.675371 17563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:06.676561 17563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:06.677868 17563 hybrid_clock.cc:648] HybridClock initialized: now 1790592306677800 us; error 51 us; skew 500 ppm
I20260928 10:45:06.678283 17563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:06 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592306, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:06.682158 17563 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:45:06.684149 17563 webserver.cc:460] Webserver started at http://127.8.218.194:36517/ using document root <none> and password file <none>
I20260928 10:45:06.685001 17563 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:06.685153 17563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:06.685521 17563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:06.688822 17563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "66e17332c357469eadbefb62fd23d72f"
format_stamp: "Formatted at 2026-09-28 10:45:06 on dist-test-slave-mn38"
server_key: "f03c11d0e8ee42e372f8eed977866747"
server_key_iv: "bc35e9c85f454e7e3cd43a61f7cafa8d"
server_key_version: "encryptionkey@0"
I20260928 10:45:06.689786 17563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "66e17332c357469eadbefb62fd23d72f"
format_stamp: "Formatted at 2026-09-28 10:45:06 on dist-test-slave-mn38"
server_key: "f03c11d0e8ee42e372f8eed977866747"
server_key_iv: "bc35e9c85f454e7e3cd43a61f7cafa8d"
server_key_version: "encryptionkey@0"
I20260928 10:45:06.696511 17563 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260928 10:45:06.701130 17578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:06.703316 17563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 10:45:06.703544 17563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "66e17332c357469eadbefb62fd23d72f"
format_stamp: "Formatted at 2026-09-28 10:45:06 on dist-test-slave-mn38"
server_key: "f03c11d0e8ee42e372f8eed977866747"
server_key_iv: "bc35e9c85f454e7e3cd43a61f7cafa8d"
server_key_version: "encryptionkey@0"
I20260928 10:45:06.703792 17563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:06.729046 17563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:06.733840 17563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:06.734109 17563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:06.735283 17563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:06.736814 17563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:06.736902 17563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:06.736991 17563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:06.737041 17563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:06.768914 17563 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:34129
I20260928 10:45:06.768970 17690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:34129 every 8 connection(s)
I20260928 10:45:06.770241 17563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:45:06.775493 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 17563
I20260928 10:45:06.775696 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:45:06.776199 9067 external_mini_cluster.cc:1428] Setting key da163bfac2c468c958d2c4f35dac4d6d
Sep 28 10:45:06 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592306, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:45:06.797040 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:06.773481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:36009)
0928 10:45:06.774250 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:06.774295 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:06.776568 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:06.777413 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:06.777423 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:06.778073 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:06.779020 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:06.779053 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:06.780275 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:06.780280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:06.782282 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:06.782305 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:06.783339 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:06.783346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:06.783676 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:06.784532 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:06.784563 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:06.787508 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:06.792220 (+ 4712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:06.792233 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:06.792255 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:06.792776 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:06.793281 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:06.793303 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:06.793308 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:06.793648 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:06.794219 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:06.794229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:06.794727 (+ 498us) client_negotiation.cc:769] Sending connection context
0928 10:45:06.795068 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 10:45:06.795712 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":306,"threads_started":1}
I20260928 10:45:06.798179 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:06.772825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:36009 (local address 127.8.218.254:42791)
0928 10:45:06.773237 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:06.773242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:06.774455 (+ 1213us) server_negotiation.cc:407] Connection header received
0928 10:45:06.776945 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:06.776951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:06.777040 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:06.777136 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:06.779328 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:06.780083 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:06.782522 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:06.783045 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:06.783181 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:06.787902 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:06.787911 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:06.787916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:06.787970 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:06.791964 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:06.792962 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:06.792969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:06.792973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:06.793144 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:06.793771 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:06.793776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:06.793780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:06.794068 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:06.794205 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:06.797685 (+ 3480us) server_negotiation.cc:299] Negotiation successful
0928 10:45:06.797900 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":195,"threads_started":1}
I20260928 10:45:06.799436 17691 heartbeater.cc:347] Connected to a master server at 127.8.218.254:42791
I20260928 10:45:06.799844 17691 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:06.800859 17691 heartbeater.cc:511] Master 127.8.218.254:42791 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:45:06.802909 17368 ts_manager.cc:194] Registered new tserver with Master: 66e17332c357469eadbefb62fd23d72f (127.8.218.194:34129)
I20260928 10:45:06.803803 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:36009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:45:06.834252 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42791
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:07.189026 17697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:07.189508 17697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:07.189675 17697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:07.197762 17697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:07.197901 17697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:07.198050 17697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:45:07.208775 17697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:07.210284 17697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:07.211716 17697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:07.226419 17702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:07.226397 17703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:07.226372 17705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:07.227401 17697 server_base.cc:1034] running on GCE node
I20260928 10:45:07.228067 17697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:07.229190 17697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:07.230446 17697 hybrid_clock.cc:648] HybridClock initialized: now 1790592307230367 us; error 73 us; skew 500 ppm
I20260928 10:45:07.230790 17697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:07 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592307, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:07.235427 17697 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:45:07.237352 17697 webserver.cc:460] Webserver started at http://127.8.218.195:43165/ using document root <none> and password file <none>
I20260928 10:45:07.238198 17697 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:07.238335 17697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:07.238673 17697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:07.242300 17697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "1c1f66e642fb4be2bae3165e5608cede"
format_stamp: "Formatted at 2026-09-28 10:45:07 on dist-test-slave-mn38"
server_key: "5ef5effe81d168c325bdb9c8eef1ee8b"
server_key_iv: "6f471a902ddb6fd7460133b24dfb6adf"
server_key_version: "encryptionkey@0"
I20260928 10:45:07.243275 17697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "1c1f66e642fb4be2bae3165e5608cede"
format_stamp: "Formatted at 2026-09-28 10:45:07 on dist-test-slave-mn38"
server_key: "5ef5effe81d168c325bdb9c8eef1ee8b"
server_key_iv: "6f471a902ddb6fd7460133b24dfb6adf"
server_key_version: "encryptionkey@0"
I20260928 10:45:07.249738 17697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:45:07.254379 17712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:07.254441 17557 heartbeater.cc:503] Master 127.8.218.254:42791 was elected leader, sending a full tablet report...
I20260928 10:45:07.256486 17697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:45:07.256714 17697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "1c1f66e642fb4be2bae3165e5608cede"
format_stamp: "Formatted at 2026-09-28 10:45:07 on dist-test-slave-mn38"
server_key: "5ef5effe81d168c325bdb9c8eef1ee8b"
server_key_iv: "6f471a902ddb6fd7460133b24dfb6adf"
server_key_version: "encryptionkey@0"
I20260928 10:45:07.256960 17697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:07.284988 17697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:07.290443 17697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:07.290732 17697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:07.291781 17697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:07.293344 17697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:07.293457 17697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:07.293579 17697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:07.293628 17697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:07.326064 17697 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:34233
I20260928 10:45:07.326120 17824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:34233 every 8 connection(s)
I20260928 10:45:07.327491 17697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:45:07.328240 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 17697
I20260928 10:45:07.328433 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:45:07.329085 9067 external_mini_cluster.cc:1428] Setting key 74dfc5d4abfb42e90f9793e2c4dbc4a1
Sep 28 10:45:07 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592307, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:45:07.350509 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:07.330512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:38509 (local address 127.8.218.254:42791)
0928 10:45:07.330977 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:07.330981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:07.332527 (+ 1546us) server_negotiation.cc:407] Connection header received
0928 10:45:07.334589 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:07.334594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:07.334671 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:07.334768 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:07.336600 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.337397 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.339093 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.339550 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.339660 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:07.343376 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:07.343384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:07.343388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:07.343438 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:07.346559 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:07.347559 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:07.347564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:07.347566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:07.347705 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:07.348390 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:07.348394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:07.348397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:07.348742 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:07.348891 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:07.349889 (+ 998us) server_negotiation.cc:299] Negotiation successful
0928 10:45:07.350104 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":233,"threads_started":1}
I20260928 10:45:07.351507 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:07.331349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:38509)
0928 10:45:07.332342 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:07.332403 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:07.334280 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:07.335066 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:07.335075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:07.335659 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:07.336388 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:07.336413 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.337589 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.337592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:07.338902 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:07.338917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.339761 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.339766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:07.340045 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:07.340694 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:07.340718 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:07.343069 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:07.346822 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:07.346832 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:07.346851 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:07.347391 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:07.347860 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:07.347864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:07.347867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:07.348234 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:07.348931 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:07.348939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:07.349363 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 10:45:07.349726 (+ 363us) client_negotiation.cc:240] Negotiation successful
0928 10:45:07.350198 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":333,"threads_started":1}
I20260928 10:45:07.353257 17825 heartbeater.cc:347] Connected to a master server at 127.8.218.254:42791
I20260928 10:45:07.353615 17825 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:07.354352 17825 heartbeater.cc:511] Master 127.8.218.254:42791 requested a full tablet report, sending...
I20260928 10:45:07.355872 17368 ts_manager.cc:194] Registered new tserver with Master: 1c1f66e642fb4be2bae3165e5608cede (127.8.218.195:34233)
I20260928 10:45:07.356850 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:38509
I20260928 10:45:07.357930 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab.
Entry for principal oryx/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab.
WARNING: no policy specified for oryx/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab.
Entry for principal oryx/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab.
WARNING: no policy specified for oryx/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab.
Entry for principal oryx/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab.
I20260928 10:45:07.534077 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 17429
Sep 28 10:45:07 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:45:07.553673 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:07.535272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.8.218.193:41565)
0928 10:45:07.537252 (+ 1980us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:07.537260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:07.537360 (+ 100us) server_negotiation.cc:407] Connection header received
0928 10:45:07.537584 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:07.537592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:07.537836 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:07.537992 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:07.539492 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.540564 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.542084 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.542704 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.542843 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:07.547450 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:07.547468 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:07.547485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:07.547541 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:07.550518 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:07.551332 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:07.551340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:07.551343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:07.551503 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:07.551895 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:07.551899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:07.551901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:07.552280 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:07.552584 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:07.552900 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 10:45:07.553179 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1804,"thread_start_us":1625,"threads_started":1}
I20260928 10:45:07.555158 17552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:07.646322 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 17429
I20260928 10:45:07.660812 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 17563
Sep 28 10:45:07 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:45:07.678596 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:07.661775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.8.218.194:34129)
0928 10:45:07.662412 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:07.662420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:07.662447 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:45:07.662507 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:07.662513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:07.662726 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:07.662869 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:07.664179 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.665284 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.666782 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.667373 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.667484 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:07.671942 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:07.671960 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:07.671978 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:07.672090 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:07.675216 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:07.676107 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:07.676117 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:07.676119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:07.676296 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:07.676891 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:07.676896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:07.676898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:07.677301 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:07.677643 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:07.677946 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 10:45:07.678180 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":332,"threads_started":1}
I20260928 10:45:07.680236 17686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:07.764242 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 17563
I20260928 10:45:07.777302 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 17697
Sep 28 10:45:07 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:45:07.796128 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:07.778259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.8.218.195:34233)
0928 10:45:07.778808 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:07.778817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:07.778848 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:45:07.778922 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:07.778929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:07.779174 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:07.779331 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:07.780716 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.781931 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.783339 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:07.784008 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:07.784141 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:07.789188 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:07.789209 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:07.789229 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:07.789290 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:07.792408 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:07.793470 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:07.793484 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:07.793495 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:07.793680 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:07.794145 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:07.794149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:07.794152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:07.794651 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:07.795020 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:07.795459 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 10:45:07.795708 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":237,"threads_started":1}
I20260928 10:45:07.797640 17820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:07.880538 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 17697
I20260928 10:45:07.894950 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 17336
I20260928 10:45:07.896106 17398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:08.002774 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 17336
I20260928 10:45:08.021268 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:42791
--webserver_interface=127.8.218.254
--webserver_port=41657
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:08.374290 17843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:08.374596 17843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:08.374687 17843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:08.382129 17843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:45:08.382266 17843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:08.382314 17843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:08.382369 17843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:45:08.382412 17843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:45:08.393115 17843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:42791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:08.394465 17843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:08.395753 17843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:08.408175 17848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:08.408144 17849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:08.408123 17851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:08.409185 17843 server_base.cc:1034] running on GCE node
I20260928 10:45:08.410142 17843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:08.411823 17843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:08.413061 17843 hybrid_clock.cc:648] HybridClock initialized: now 1790592308412998 us; error 60 us; skew 500 ppm
I20260928 10:45:08.413553 17843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:08 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:08.416829 17843 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:45:08.418503 17843 webserver.cc:460] Webserver started at http://127.8.218.254:41657/ using document root <none> and password file <none>
I20260928 10:45:08.419212 17843 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:08.419341 17843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:08.425102 17843 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:45:08.428830 17858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:08.430754 17843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:45:08.430987 17843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "c00e1edf3fd640208aa08007fc26afd0"
format_stamp: "Formatted at 2026-09-28 10:45:05 on dist-test-slave-mn38"
server_key: "7af08941ca74b652932c721e3487d4f6"
server_key_iv: "2e000bb8a206682629aa9738a7753ade"
server_key_version: "encryptionkey@0"
I20260928 10:45:08.432114 17843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:08.459717 17843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:08.464892 17843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:08.465175 17843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:08.484299 17843 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:42791
I20260928 10:45:08.484323 17909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:42791 every 8 connection(s)
I20260928 10:45:08.486084 17843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:45:08.495368 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 17843
I20260928 10:45:08.495667 17910 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:45:08.496462 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:41565
--local_ip_for_outbound_sockets=127.8.218.193
--tserver_master_addrs=127.8.218.254:42791
--webserver_port=33511
--webserver_interface=127.8.218.193
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.218.193
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:45:08.499908 17910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Bootstrap starting.
I20260928 10:45:08.521060 17910 log.cc:824] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:08.531061 17910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:45:08.531597 17910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Bootstrap complete.
I20260928 10:45:08.541257 17910 raft_consensus.cc:348] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } }
I20260928 10:45:08.542222 17910 raft_consensus.cc:729] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00e1edf3fd640208aa08007fc26afd0, State: Initialized, Role: FOLLOWER
I20260928 10:45:08.542846 17910 consensus_queue.cc:260] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } }
I20260928 10:45:08.543076 17910 raft_consensus.cc:388] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:45:08.543186 17910 raft_consensus.cc:482] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:45:08.543345 17910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:45:08.547461 17910 raft_consensus.cc:504] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } }
I20260928 10:45:08.548004 17910 leader_election.cc:302] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c00e1edf3fd640208aa08007fc26afd0; no voters:
I20260928 10:45:08.548672 17910 leader_election.cc:288] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:45:08.548785 17914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:45:08.549918 17914 raft_consensus.cc:686] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [term 2 LEADER]: Becoming Leader. State: Replica: c00e1edf3fd640208aa08007fc26afd0, State: Running, Role: LEADER
I20260928 10:45:08.550724 17914 consensus_queue.cc:237] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } }
I20260928 10:45:08.550935 17910 sys_catalog.cc:564] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:45:08.553200 17916 sys_catalog.cc:455] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c00e1edf3fd640208aa08007fc26afd0. Latest consensus state: current_term: 2 leader_uuid: "c00e1edf3fd640208aa08007fc26afd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } } }
I20260928 10:45:08.553601 17916 sys_catalog.cc:458] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:08.554111 17915 sys_catalog.cc:455] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c00e1edf3fd640208aa08007fc26afd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00e1edf3fd640208aa08007fc26afd0" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 42791 } } }
I20260928 10:45:08.554407 17915 sys_catalog.cc:458] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0 [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:08.563231 17921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:45:08.570330 17921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:45:08.572950 17921 catalog_manager.cc:1259] Loaded cluster ID: c22d800c95ca45f1887fad639afba443
I20260928 10:45:08.573107 17921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:45:08.577808 17921 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:45:08.579654 17921 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c00e1edf3fd640208aa08007fc26afd0: Loaded TSK: 0
I20260928 10:45:08.581588 17921 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:45:08.868147 17912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:08.868448 17912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:08.868625 17912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:08.876127 17912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:08.876252 17912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:08.876395 17912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:45:08.887384 17912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.193.keytab
--principal=oryx/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:41565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=33511
--tserver_master_addrs=127.8.218.254:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:08.888779 17912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:08.890128 17912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:08.905296 17936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:08.905351 17937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:08.905510 17939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:08.906370 17912 server_base.cc:1034] running on GCE node
I20260928 10:45:08.907137 17912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:08.908351 17912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:08.909565 17912 hybrid_clock.cc:648] HybridClock initialized: now 1790592308909500 us; error 44 us; skew 500 ppm
I20260928 10:45:08.909982 17912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:08 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592308, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:08.913866 17912 init.cc:377] Logged in from keytab as oryx/127.8.218.193@KRBTEST.COM (short username oryx)
I20260928 10:45:08.915766 17912 webserver.cc:460] Webserver started at http://127.8.218.193:33511/ using document root <none> and password file <none>
I20260928 10:45:08.916728 17912 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:08.916899 17912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:08.924933 17912 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:45:08.929316 17946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:08.931175 17912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 10:45:08.931383 17912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "329eb79450574aa8bba9123101db61fc"
format_stamp: "Formatted at 2026-09-28 10:45:06 on dist-test-slave-mn38"
server_key: "2899058ed6a29fc017ae0c6750732ffa"
server_key_iv: "444c1da58ad9a521eae633918143b7fc"
server_key_version: "encryptionkey@0"
I20260928 10:45:08.932679 17912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:08.948001 17912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:08.952401 17912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:08.952669 17912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:08.953742 17912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:08.955199 17912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:08.955304 17912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:08.955423 17912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:08.955500 17912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:08.993335 17912 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:41565
I20260928 10:45:08.993492 18058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:41565 every 8 connection(s)
I20260928 10:45:08.995060 17912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:45:09.001206 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 17912
I20260928 10:45:09.001936 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:34129
--local_ip_for_outbound_sockets=127.8.218.194
--tserver_master_addrs=127.8.218.254:42791
--webserver_port=36517
--webserver_interface=127.8.218.194
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.218.194
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:45:09 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:09 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:45:09.357057 18063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:09.357369 18063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:09.357537 18063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:09.365104 18063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:09.365260 18063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:09.365412 18063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:45:09.375447 18063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.194.keytab
--principal=oryx/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:34129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=36517
--tserver_master_addrs=127.8.218.254:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:09.376776 18063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:09.378201 18063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:09.393981 18070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:09.394594 18069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:09.394966 18072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:09.396261 18063 server_base.cc:1034] running on GCE node
I20260928 10:45:09.396988 18063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:09.398061 18063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:09.399361 18063 hybrid_clock.cc:648] HybridClock initialized: now 1790592309399285 us; error 76 us; skew 500 ppm
I20260928 10:45:09.399695 18063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:09 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592309, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:09.403864 18063 init.cc:377] Logged in from keytab as oryx/127.8.218.194@KRBTEST.COM (short username oryx)
I20260928 10:45:09.405882 18063 webserver.cc:460] Webserver started at http://127.8.218.194:36517/ using document root <none> and password file <none>
I20260928 10:45:09.406745 18063 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:09.406891 18063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:09.413193 18063 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 10:45:09.416566 18079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:09.418743 18063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:45:09.418938 18063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "66e17332c357469eadbefb62fd23d72f"
format_stamp: "Formatted at 2026-09-28 10:45:06 on dist-test-slave-mn38"
server_key: "f03c11d0e8ee42e372f8eed977866747"
server_key_iv: "bc35e9c85f454e7e3cd43a61f7cafa8d"
server_key_version: "encryptionkey@0"
I20260928 10:45:09.420022 18063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:09.450107 18063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:09.454662 18063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:09.454945 18063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:09.456095 18063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:09.457824 18063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:09.457938 18063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:09.458062 18063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:09.458129 18063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:09.499588 18063 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:34129
I20260928 10:45:09.499625 18191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:34129 every 8 connection(s)
I20260928 10:45:09.501169 18063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:45:09.509004 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 18063
I20260928 10:45:09.509924 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:34233
--local_ip_for_outbound_sockets=127.8.218.195
--tserver_master_addrs=127.8.218.254:42791
--webserver_port=43165
--webserver_interface=127.8.218.195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.218.195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:45:09 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:09 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:45:09.877954 18196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:09.878257 18196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:09.878422 18196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:09.886219 18196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:09.886368 18196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:09.886561 18196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:45:09.896770 18196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/krb5kdc/oryx_127.8.218.195.keytab
--principal=oryx/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:34233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=43165
--tserver_master_addrs=127.8.218.254:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:09.898324 18196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:09.899742 18196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:09.918812 18203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:09.925531 18202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:09.927992 18205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:09.928205 18196 server_base.cc:1034] running on GCE node
I20260928 10:45:09.929008 18196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:09.930184 18196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:09.931530 18196 hybrid_clock.cc:648] HybridClock initialized: now 1790592309931456 us; error 66 us; skew 500 ppm
I20260928 10:45:09.931913 18196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:09 dist-test-slave-mn38 krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592309, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:09.935281 18196 init.cc:377] Logged in from keytab as oryx/127.8.218.195@KRBTEST.COM (short username oryx)
I20260928 10:45:09.937708 18196 webserver.cc:460] Webserver started at http://127.8.218.195:43165/ using document root <none> and password file <none>
I20260928 10:45:09.938602 18196 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:09.938746 18196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:09.945528 18196 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:45:09.948841 18212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:09.950614 18196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:45:09.950839 18196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "1c1f66e642fb4be2bae3165e5608cede"
format_stamp: "Formatted at 2026-09-28 10:45:07 on dist-test-slave-mn38"
server_key: "5ef5effe81d168c325bdb9c8eef1ee8b"
server_key_iv: "6f471a902ddb6fd7460133b24dfb6adf"
server_key_version: "encryptionkey@0"
I20260928 10:45:09.951957 18196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:09.975517 18196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:09.979854 18196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:09.980139 18196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:09.981216 18196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:09.982792 18196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:09.982890 18196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:09.983003 18196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:09.983069 18196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:10.039243 18196 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:34233
I20260928 10:45:10.039270 18324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:34233 every 8 connection(s)
I20260928 10:45:10.040863 18196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:45:10.048961 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 18196
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:10.080505 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:10.050550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.8.218.254:42791)
0928 10:45:10.058798 (+ 8248us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:10.058806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:10.058832 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:45:10.058910 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:10.058916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:10.059019 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:10.059164 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:10.061351 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.063042 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.064891 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.065652 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.065789 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.070203 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:10.070217 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:10.070244 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:10.070309 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:10.074359 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:10.075402 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:10.075412 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:10.075423 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:10.075660 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:10.076211 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:10.076216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:10.076220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:10.076939 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:10.078512 (+ 1573us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:10.078832 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 10:45:10.079365 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8069,"thread_start_us":7908,"threads_started":1}
W20260928 10:45:10.231277 18062 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.221s user 0.001s sys 0.015s
W20260928 10:45:10.233397 18061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:08.998494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:46583 (local address 127.8.218.254:42791)
0928 10:45:08.999265 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:08.999278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:08.999652 (+ 374us) server_negotiation.cc:407] Connection header received
0928 10:45:09.002049 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:09.002071 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:09.002609 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:09.003087 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:09.004819 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:09.006607 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:09.008297 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:09.009081 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:09.009225 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.232950 (+1223725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:46583: BlockingRecv error: recv got EOF from 127.8.218.193:46583 (error 108)
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":298,"threads_started":1}
I20260928 10:45:10.234102 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:08.998633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:46583)
0928 10:45:08.999476 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:08.999521 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:09.001283 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:09.003285 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:09.003295 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:09.003929 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:09.004550 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:09.004570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:09.006809 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:09.006814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:09.008135 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:09.008149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:09.009361 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:09.009369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:09.009720 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:09.010488 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:09.010518 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:10.232495 (+1221977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":351,"threads_started":1}
W20260928 10:45:10.235407 18059 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:42791 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.218.254:42791: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:45:10.253523 18061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:10.236548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:47153 (local address 127.8.218.254:42791)
0928 10:45:10.236777 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:10.236785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:10.236927 (+ 142us) server_negotiation.cc:407] Connection header received
0928 10:45:10.237174 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:10.237181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:10.237315 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:10.237474 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:10.238957 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.240482 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.244169 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.244968 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.245195 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.253166 (+ 7971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:47153: BlockingRecv error: recv got EOF from 127.8.218.193:47153 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:45:10.254101 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:10.236449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:47153)
0928 10:45:10.236805 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:10.236831 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:10.237018 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:10.237544 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:10.237550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:10.238254 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:10.238628 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.238648 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.240877 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.240886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.243882 (+ 2996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.243924 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.245284 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.245292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.245583 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.246496 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:10.246526 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:10.252736 (+ 6210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190}
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:45:10.271661 18061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:10.255658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:59471 (local address 127.8.218.254:42791)
0928 10:45:10.255865 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:10.255872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:10.256158 (+ 286us) server_negotiation.cc:407] Connection header received
0928 10:45:10.256595 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:10.256602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:10.256726 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:10.256823 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:10.258596 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.260002 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.262494 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.263294 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.263459 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.271303 (+ 7844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:59471: BlockingRecv error: recv got EOF from 127.8.218.193:59471 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:45:10.272764 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:10.255195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:59471)
0928 10:45:10.256008 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:10.256033 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:10.256350 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:10.257111 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:10.257117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:10.257965 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:10.258354 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.258369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.260286 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.260293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.262164 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.262190 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.263975 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.263984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.264224 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.265405 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:10.265462 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:10.272275 (+ 6813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":645,"threads_started":1}
W20260928 10:45:10.273237 18059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:45:10.739192 18195 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.224s user 0.005s sys 0.009s
W20260928 10:45:10.740622 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:09.504393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:41819 (local address 127.8.218.254:42791)
0928 10:45:09.504835 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:09.504839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:09.506401 (+ 1562us) server_negotiation.cc:407] Connection header received
0928 10:45:09.508387 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:09.508393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:09.508494 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:09.508591 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:09.510452 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:09.511263 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:09.513125 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:09.513655 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:09.513780 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.740294 (+1226514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:41819: BlockingRecv error: recv got EOF from 127.8.218.194:41819 (error 108)
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":223,"threads_started":1}
I20260928 10:45:10.741235 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:09.505300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:41819)
0928 10:45:09.506226 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:09.506271 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:09.508052 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:09.508884 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:09.508894 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:09.509533 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:09.510242 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:09.510269 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:09.511473 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:09.511477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:09.512928 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:09.512946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:09.513917 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:09.513925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:09.514182 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:09.514895 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:09.514919 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:10.740013 (+1225094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":365,"threads_started":1}
W20260928 10:45:10.742280 18192 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:42791 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.218.254:42791: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:10.759598 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:10.742953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:60451)
0928 10:45:10.743213 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:10.743240 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:10.743399 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:10.743739 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:10.743745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:10.744386 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:10.744640 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.744656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.746008 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.746015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.747787 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.747810 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.748820 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.748828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.749050 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.754532 (+ 5482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:10.754570 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:10.759232 (+ 4662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 10:45:10.759665 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:10.743040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:60451 (local address 127.8.218.254:42791)
0928 10:45:10.743413 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:10.743420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:10.743442 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:45:10.743515 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:10.743520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:10.743601 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:10.743681 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:10.744857 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.745845 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.748007 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.748550 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.748732 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.759455 (+ 10723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:60451: BlockingRecv error: recv got EOF from 127.8.218.194:60451 (error 108)
Metrics: {"server-negotiator.queue_time_us":260}
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:10 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:10.770249 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:10.760352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:60297)
0928 10:45:10.760521 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:10.760539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:10.760646 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:10.761198 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:10.761202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:10.761554 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:10.761762 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.761771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.763379 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.763383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.764483 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.764498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.765543 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.765548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.765746 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.766360 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:10.766382 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:10.770026 (+ 3644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 10:45:10.770699 18192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:45:10.770696 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:10.760616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:60297 (local address 127.8.218.254:42791)
0928 10:45:10.760818 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:10.760826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:10.760847 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:45:10.760917 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:10.760924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:10.761023 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:10.761149 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:10.761897 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.763190 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.764626 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.765347 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.765548 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.770378 (+ 4830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:60297: BlockingRecv error: recv got EOF from 127.8.218.194:60297 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 10:45:11.123950 18327 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.058s user 0.004s sys 0.012s
W20260928 10:45:11.125357 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:10.046296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:36541 (local address 127.8.218.254:42791)
0928 10:45:10.052124 (+ 5828us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:10.052131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:10.052156 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:45:10.052228 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:10.052234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:10.052341 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:10.052470 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:10.058967 (+ 6497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.060316 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.063503 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.064254 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.064426 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.125034 (+1060608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:36541: BlockingRecv error: recv got EOF from 127.8.218.195:36541 (error 108)
Metrics: {"server-negotiator.queue_time_us":5689,"thread_start_us":5351,"threads_started":1}
I20260928 10:45:11.125774 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:10.045431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:36541)
0928 10:45:10.048033 (+ 2602us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:10.048089 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:10.050461 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:10.055996 (+ 5535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:10.056015 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:10.057556 (+ 1541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:10.058681 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.058719 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.060617 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.060623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.063290 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:10.063326 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:10.064581 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:10.064590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:10.064992 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:10.065956 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:10.065998 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:11.124732 (+1058734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2210,"thread_start_us":1931,"threads_started":1}
W20260928 10:45:11.126658 18325 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:42791 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.218.254:42791: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Sep 28 10:45:11 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:11 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:11.138381 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:11.127327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:51447)
0928 10:45:11.127552 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:11.127573 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:11.127707 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:11.128051 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:11.128056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:11.128569 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:11.128789 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:11.128802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.130092 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.130098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:11.131707 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:11.131729 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.132686 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.132693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:11.132888 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.133571 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:11.133597 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:11.138118 (+ 4521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 10:45:11.138662 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:11.127455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:51447 (local address 127.8.218.254:42791)
0928 10:45:11.127621 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:11.127626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:11.127644 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:45:11.127858 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:11.127862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:11.127924 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:11.127999 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:11.128934 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.129940 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.131895 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.132482 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.132595 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.138420 (+ 5825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:51447: BlockingRecv error: recv got EOF from 127.8.218.195:51447 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 10:45:11 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:11 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:11.149514 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:11.139333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:59979)
0928 10:45:11.139669 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:11.139687 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:11.139788 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:11.140181 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:11.140185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:11.140501 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:11.140746 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:11.140756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.142269 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.142273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:11.143533 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:11.143550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.144649 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.144653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:11.144805 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.145394 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:11.145413 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:11.149246 (+ 3833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147}
W20260928 10:45:11.149849 18325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:45:11.149874 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:11.139426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:59979 (local address 127.8.218.254:42791)
0928 10:45:11.139718 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:11.139725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:11.139749 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:45:11.139903 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:11.139908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:11.140013 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:11.140126 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:11.140899 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.142077 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.143669 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.144276 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.144396 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.149577 (+ 5181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:59979: BlockingRecv error: recv got EOF from 127.8.218.195:59979 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Sep 28 10:45:11 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:11 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:11.284438 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:11.273894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:58451)
0928 10:45:11.274423 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:11.274440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:11.274559 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:11.274903 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:11.274906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:11.275265 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:11.275416 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:11.275425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.276872 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.276876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:11.277874 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:11.277886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.278884 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.278892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:11.279118 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.279817 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:11.279871 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:11.284135 (+ 4264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":323,"threads_started":1}
W20260928 10:45:11.284862 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:11.273985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:58451 (local address 127.8.218.254:42791)
0928 10:45:11.274186 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:11.274193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:11.274511 (+ 318us) server_negotiation.cc:407] Connection header received
0928 10:45:11.274664 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:11.274669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:11.274752 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:11.274863 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:11.275543 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.276709 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.278007 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.278677 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.278805 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.284609 (+ 5804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:58451: BlockingRecv error: recv got EOF from 127.8.218.193:58451 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 10:45:11 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:11 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:11.781068 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:11.771394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:60309)
0928 10:45:11.771964 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:11.771980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:11.772105 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:11.772496 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:11.772499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:11.772959 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:11.773106 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:11.773114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.774357 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.774362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:11.775437 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:11.775452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.776454 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.776461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:11.776653 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.777300 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:11.777319 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:11.780812 (+ 3493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":335,"threads_started":1}
W20260928 10:45:11.781301 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:11.771550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:60309 (local address 127.8.218.254:42791)
0928 10:45:11.771856 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:11.771862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:11.772064 (+ 202us) server_negotiation.cc:407] Connection header received
0928 10:45:11.772257 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:11.772262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:11.772335 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:11.772425 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:11.773300 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.774181 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.775634 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:11.776253 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:11.776394 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:11.781051 (+ 4657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:60309: BlockingRecv error: recv got EOF from 127.8.218.194:60309 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Sep 28 10:45:12 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:12 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:12.159799 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:12.150505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:44137)
0928 10:45:12.151073 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:12.151089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:12.151207 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:12.151570 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:12.151573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:12.152027 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:12.152177 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:12.152186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.153116 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.153120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:12.154105 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:12.154118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.155061 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.155068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:12.155230 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:12.155955 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:12.155973 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:12.159557 (+ 3584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":326,"threads_started":1}
W20260928 10:45:12.159957 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:12.150600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:44137 (local address 127.8.218.254:42791)
0928 10:45:12.150809 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:12.150816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:12.151163 (+ 347us) server_negotiation.cc:407] Connection header received
0928 10:45:12.151326 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:12.151330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:12.151416 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:12.151524 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:12.152308 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.153001 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.154258 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.154825 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.154964 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:12.159768 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:44137: BlockingRecv error: recv got EOF from 127.8.218.195:44137 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 10:45:12 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:12 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:12.294932 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:12.285586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:33227)
0928 10:45:12.286099 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:12.286120 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:12.286252 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:12.286552 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:12.286554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:12.286997 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:12.287182 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:12.287193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.288230 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.288235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:12.289238 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:12.289254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.290138 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.290144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:12.290307 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:12.290932 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:12.290951 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:12.294616 (+ 3665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":257,"threads_started":1}
W20260928 10:45:12.295204 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:12.285620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:33227 (local address 127.8.218.254:42791)
0928 10:45:12.285850 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:12.285860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:12.286207 (+ 347us) server_negotiation.cc:407] Connection header received
0928 10:45:12.286366 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:12.286370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:12.286437 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:12.286514 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:12.287342 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.288086 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.289406 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.289935 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.290050 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:12.294932 (+ 4882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:33227: BlockingRecv error: recv got EOF from 127.8.218.193:33227 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 10:45:12 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:12 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:12.793942 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:12.782193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:57809)
0928 10:45:12.782737 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:12.782759 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:12.782882 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:12.783218 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:12.783221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:12.783687 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:12.783881 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:12.783892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.785020 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.785025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:12.786047 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:12.786062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.786871 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.786876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:12.787084 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:12.787802 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:12.787830 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:12.793543 (+ 5713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":305,"threads_started":1}
W20260928 10:45:12.794245 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:12.782264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:57809 (local address 127.8.218.254:42791)
0928 10:45:12.782539 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:12.782545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:12.782847 (+ 302us) server_negotiation.cc:407] Connection header received
0928 10:45:12.782999 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:12.783004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:12.783073 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:12.783165 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:12.784033 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.784833 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.786217 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:12.786704 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:12.786815 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:12.794002 (+ 7187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:57809: BlockingRecv error: recv got EOF from 127.8.218.194:57809 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 28 10:45:13 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:13 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:13.170475 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:13.160879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:34973)
0928 10:45:13.161387 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:13.161407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:13.161608 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:13.161961 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:13.161965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:13.162502 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:13.162739 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:13.162754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.164003 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.164009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:13.165050 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:13.165067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.165877 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.165883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:13.166072 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:13.166668 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:13.166687 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:13.170133 (+ 3446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
W20260928 10:45:13.170594 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:13.161182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:34973 (local address 127.8.218.254:42791)
0928 10:45:13.161394 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:13.161401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:13.161515 (+ 114us) server_negotiation.cc:407] Connection header received
0928 10:45:13.161735 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:13.161740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:13.161801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:13.161898 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:13.162904 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.163870 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.165217 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.165702 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.165830 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:13.170348 (+ 4518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:34973: BlockingRecv error: recv got EOF from 127.8.218.195:34973 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 10:45:13 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:13 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:13.305334 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:13.296230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:41617)
0928 10:45:13.296782 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:13.296800 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:13.296947 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:13.297287 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:13.297291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:13.297711 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:13.297882 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:13.297891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.298948 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.298953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:13.299954 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:13.299970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.300722 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.300727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:13.300903 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:13.301543 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:13.301563 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:13.305106 (+ 3543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":296,"threads_started":1}
W20260928 10:45:13.305546 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:13.296444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:41617 (local address 127.8.218.254:42791)
0928 10:45:13.296690 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:13.296698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:13.296890 (+ 192us) server_negotiation.cc:407] Connection header received
0928 10:45:13.297071 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:13.297077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:13.297148 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:13.297245 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:13.298035 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.298794 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.300117 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.300561 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.300680 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:13.305322 (+ 4642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:41617: BlockingRecv error: recv got EOF from 127.8.218.193:41617 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 28 10:45:13 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:13 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:13.804944 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:13.795401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:58037)
0928 10:45:13.795896 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:13.795912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:13.796047 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:13.796387 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:13.796390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:13.796761 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:13.796928 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:13.796938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.798077 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.798082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:13.798977 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:13.798990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.799767 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.799771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:13.799904 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:13.800516 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:13.800534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:13.804679 (+ 4145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":297,"threads_started":1}
W20260928 10:45:13.805143 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:13.795507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:58037 (local address 127.8.218.254:42791)
0928 10:45:13.795731 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:13.795738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:13.795996 (+ 258us) server_negotiation.cc:407] Connection header received
0928 10:45:13.796163 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:13.796168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:13.796235 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:13.796319 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:13.797085 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.797849 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.799143 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:13.799614 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:13.799718 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:13.804895 (+ 5177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:58037: BlockingRecv error: recv got EOF from 127.8.218.194:58037 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 10:45:14 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:14 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:14.180461 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:14.171517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:44203)
0928 10:45:14.172035 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:14.172058 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:14.172191 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:14.172503 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:14.172508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:14.172892 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:14.173070 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:14.173080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.174135 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.174141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:14.175063 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:14.175076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.175756 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.175762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:14.175930 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:14.176567 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:14.176593 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:14.180218 (+ 3625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
W20260928 10:45:14.180692 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:14.171599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:44203 (local address 127.8.218.254:42791)
0928 10:45:14.171779 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:14.171785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:14.172146 (+ 361us) server_negotiation.cc:407] Connection header received
0928 10:45:14.172305 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:14.172310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:14.172380 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:14.172458 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:14.173219 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.173996 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.175204 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.175617 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.175727 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:14.180459 (+ 4732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:44203: BlockingRecv error: recv got EOF from 127.8.218.195:44203 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 10:45:14 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:14 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:14.315889 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:14.306514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:51647)
0928 10:45:14.307001 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:14.307018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:14.307143 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:14.307476 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:14.307480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:14.307841 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:14.308003 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:14.308013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.309182 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.309186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:14.310216 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:14.310230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.311067 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.311074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:14.311267 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:14.311981 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:14.312004 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:14.315622 (+ 3618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":272,"threads_started":1}
W20260928 10:45:14.316108 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:14.306702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:51647 (local address 127.8.218.254:42791)
0928 10:45:14.306891 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:14.306895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:14.307117 (+ 222us) server_negotiation.cc:407] Connection header received
0928 10:45:14.307244 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:14.307248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:14.307313 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:14.307408 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:14.308188 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.308970 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.310376 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.310912 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.311037 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:14.315879 (+ 4842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:51647: BlockingRecv error: recv got EOF from 127.8.218.193:51647 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 10:45:14 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:14 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:14.817845 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:14.806129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:59735)
0928 10:45:14.806797 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:14.806817 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:14.807049 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:14.807482 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:14.807486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:14.807868 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:14.808038 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:14.808048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.809604 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.809610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:14.811523 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:14.811552 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.812965 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.812970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:14.813125 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:14.813879 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:14.813904 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:14.817577 (+ 3673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":406,"threads_started":1}
W20260928 10:45:14.818038 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:14.806564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:59735 (local address 127.8.218.254:42791)
0928 10:45:14.806755 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:14.806762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:14.806910 (+ 148us) server_negotiation.cc:407] Connection header received
0928 10:45:14.807207 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:14.807214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:14.807303 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:14.807427 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:14.808194 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.809430 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.811902 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:14.812739 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:14.812878 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:14.817812 (+ 4934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:59735: BlockingRecv error: recv got EOF from 127.8.218.194:59735 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 10:45:15 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:15 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:15.191403 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:15.181672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:35267)
0928 10:45:15.182254 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:15.182274 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:15.182440 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:15.182801 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:15.182805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:15.183175 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:15.183335 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:15.183345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.184393 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.184397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:15.185417 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:15.185475 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.186235 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.186241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:15.186422 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:15.187014 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:15.187033 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:15.191028 (+ 3995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":335,"threads_started":1}
W20260928 10:45:15.191558 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:15.181774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:35267 (local address 127.8.218.254:42791)
0928 10:45:15.182052 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:15.182059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:15.182383 (+ 324us) server_negotiation.cc:407] Connection header received
0928 10:45:15.182569 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:15.182573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:15.182635 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:15.182718 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:15.183495 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.184246 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.185619 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.186064 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.186189 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:15.191332 (+ 5143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:35267: BlockingRecv error: recv got EOF from 127.8.218.195:35267 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 10:45:15 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:15 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:15.326287 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:15.317002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:51447)
0928 10:45:15.317539 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:15.317568 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:15.317707 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:15.318030 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:15.318035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:15.318502 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:15.318670 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:15.318678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.319709 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.319713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:15.320716 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:15.320733 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.321514 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.321521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:15.321697 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:15.322377 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:15.322413 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:15.325963 (+ 3550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
W20260928 10:45:15.326467 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:15.317103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:51447 (local address 127.8.218.254:42791)
0928 10:45:15.317336 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:15.317341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:15.317654 (+ 313us) server_negotiation.cc:407] Connection header received
0928 10:45:15.317821 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:15.317826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:15.317889 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:15.317978 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:15.318815 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.319549 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.320895 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.321305 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.321465 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:15.326192 (+ 4727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:51447: BlockingRecv error: recv got EOF from 127.8.218.193:51447 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 10:45:15 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:15 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:15.835141 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:15.819125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:53117)
0928 10:45:15.819629 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:15.819651 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:15.819901 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:15.820325 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:15.820330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:15.820824 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:15.820979 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:15.820988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.822238 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.822243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:15.823483 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:15.823498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.824333 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.824339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:15.824538 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:15.831031 (+ 6493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:15.831057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:15.834794 (+ 3737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":228,"threads_started":1}
W20260928 10:45:15.835251 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:15.819415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:53117 (local address 127.8.218.254:42791)
0928 10:45:15.819661 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:15.819667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:15.819762 (+ 95us) server_negotiation.cc:407] Connection header received
0928 10:45:15.820089 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:15.820093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:15.820160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:15.820246 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:15.821154 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.822091 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.823709 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:15.824184 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:15.824288 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:15.835005 (+ 10717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:53117: BlockingRecv error: recv got EOF from 127.8.218.194:53117 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 28 10:45:16 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:16 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:16.201889 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:16.192481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:47237)
0928 10:45:16.192984 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:16.193003 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:16.193149 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:16.193502 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:16.193507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:16.193952 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:16.194103 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:16.194113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.195181 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.195186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:16.196346 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:16.196364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.197172 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.197179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:16.197353 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:16.198047 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:16.198069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:16.201490 (+ 3421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":327,"threads_started":1}
W20260928 10:45:16.201969 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:16.192635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:47237 (local address 127.8.218.254:42791)
0928 10:45:16.192820 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:16.192826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:16.193109 (+ 283us) server_negotiation.cc:407] Connection header received
0928 10:45:16.193266 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:16.193272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:16.193341 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:16.193421 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:16.194265 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.195013 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.196556 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.196993 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.197136 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:16.201757 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:47237: BlockingRecv error: recv got EOF from 127.8.218.195:47237 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 10:45:16 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:16 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:16.337728 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:16.327586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:51839)
0928 10:45:16.328242 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:16.328263 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:16.328402 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:16.328808 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:16.328813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:16.329239 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:16.329420 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:16.329430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.330667 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.330674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:16.331823 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:16.331844 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.332787 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.332792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:16.332937 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:16.333634 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:16.333654 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:16.337341 (+ 3687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":403,"threads_started":1}
W20260928 10:45:16.337929 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:16.327732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:51839 (local address 127.8.218.254:42791)
0928 10:45:16.327973 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:16.327978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:16.328347 (+ 369us) server_negotiation.cc:407] Connection header received
0928 10:45:16.328529 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:16.328533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:16.328607 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:16.328726 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:16.329630 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.330465 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.331998 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.332569 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.332762 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:16.337663 (+ 4901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:51839: BlockingRecv error: recv got EOF from 127.8.218.193:51839 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 10:45:16 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:16 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:16.845115 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:16.836093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:38781)
0928 10:45:16.836676 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:16.836693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:16.836829 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:16.837194 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:16.837198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:16.837668 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:16.837833 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:16.837842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.838948 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.838952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:16.839889 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:16.839903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.840661 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.840667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:16.840839 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:16.841462 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:16.841487 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:16.844871 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":230,"threads_started":1}
W20260928 10:45:16.845271 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:16.836159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:38781 (local address 127.8.218.254:42791)
0928 10:45:16.836361 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:16.836367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:16.836778 (+ 411us) server_negotiation.cc:407] Connection header received
0928 10:45:16.836940 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:16.836945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:16.837012 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:16.837106 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:16.837961 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.838829 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.840036 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:16.840483 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:16.840583 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:16.845071 (+ 4488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:38781: BlockingRecv error: recv got EOF from 127.8.218.194:38781 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 10:45:17 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:17 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:17.211927 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:17.202880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:48519)
0928 10:45:17.203351 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:17.203367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:17.203500 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:17.203836 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:17.203841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:17.204264 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:17.204430 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:17.204440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.205504 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.205508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:17.206436 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:17.206449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.207243 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.207249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:17.207411 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:17.208001 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:17.208020 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:17.211513 (+ 3493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
W20260928 10:45:17.212010 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:17.202988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:48519 (local address 127.8.218.254:42791)
0928 10:45:17.203218 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:17.203225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:17.203445 (+ 220us) server_negotiation.cc:407] Connection header received
0928 10:45:17.203614 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:17.203619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:17.203684 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:17.203779 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:17.204580 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.205316 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.206597 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.207074 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.207203 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:17.211773 (+ 4570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:48519: BlockingRecv error: recv got EOF from 127.8.218.195:48519 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 10:45:17 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:17 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:17.347954 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:17.338800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:58339)
0928 10:45:17.339372 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:17.339388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:17.339512 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:17.339776 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:17.339779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:17.340197 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:17.340383 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:17.340392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.341363 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.341368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:17.342349 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:17.342361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.343074 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.343078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:17.343214 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:17.343750 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:17.343768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:17.347711 (+ 3943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":263,"threads_started":1}
W20260928 10:45:17.348207 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:17.338822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:58339 (local address 127.8.218.254:42791)
0928 10:45:17.339008 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:17.339013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:17.339458 (+ 445us) server_negotiation.cc:407] Connection header received
0928 10:45:17.339606 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:17.339609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:17.339671 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:17.339750 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:17.340514 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.341224 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.342482 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.342940 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.343042 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:17.347979 (+ 4937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:58339: BlockingRecv error: recv got EOF from 127.8.218.193:58339 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 10:45:17 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:17 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:17.855840 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:17.846213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:58385)
0928 10:45:17.846884 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:17.846902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:17.847031 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:17.847452 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:17.847457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:17.847857 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:17.848036 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:17.848045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.849183 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.849188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:17.850272 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:17.850286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.851289 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.851295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:17.851429 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:17.852020 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:17.852038 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:17.855494 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":264,"threads_started":1}
W20260928 10:45:17.856010 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:17.846312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:58385 (local address 127.8.218.254:42791)
0928 10:45:17.846492 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:17.846496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:17.846988 (+ 492us) server_negotiation.cc:407] Connection header received
0928 10:45:17.847186 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:17.847190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:17.847263 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:17.847360 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:17.848176 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.849017 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.850405 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:17.851118 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:17.851228 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:17.855734 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:58385: BlockingRecv error: recv got EOF from 127.8.218.194:58385 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 10:45:18 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:18 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:18.222498 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:18.213097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:42255)
0928 10:45:18.213798 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:18.213820 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:18.213941 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:18.214221 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:18.214225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:18.214592 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:18.214770 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:18.214779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.215874 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.215878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:18.216885 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:18.216899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.217670 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.217675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:18.217805 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:18.218397 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:18.218415 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:18.222130 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":288,"threads_started":1}
W20260928 10:45:18.222553 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:18.213192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:42255 (local address 127.8.218.254:42791)
0928 10:45:18.213375 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:18.213380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:18.213904 (+ 524us) server_negotiation.cc:407] Connection header received
0928 10:45:18.214035 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:18.214039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:18.214109 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:18.214193 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:18.214924 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.215731 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.217047 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.217509 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.217611 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:18.222347 (+ 4736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:42255: BlockingRecv error: recv got EOF from 127.8.218.195:42255 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 10:45:18 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:18 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:18.357949 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:18.348845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:34203)
0928 10:45:18.349471 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:18.349490 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:18.349618 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:18.349924 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:18.349929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:18.350332 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:18.350504 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:18.350513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.351615 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.351621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:18.352602 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:18.352615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.353365 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.353370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:18.353567 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:18.354205 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:18.354224 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:18.357609 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":342,"threads_started":1}
W20260928 10:45:18.358068 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:18.349046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:34203 (local address 127.8.218.254:42791)
0928 10:45:18.349270 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:18.349275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:18.349566 (+ 291us) server_negotiation.cc:407] Connection header received
0928 10:45:18.349727 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:18.349732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:18.349798 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:18.349877 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:18.350650 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.351479 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.352738 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.353188 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.353309 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:18.357845 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:34203: BlockingRecv error: recv got EOF from 127.8.218.193:34203 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 10:45:18 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:18 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 10:45:18.867327 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:18.857080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:60351 (local address 127.8.218.254:42791)
0928 10:45:18.857229 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:18.857233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:18.857665 (+ 432us) server_negotiation.cc:407] Connection header received
0928 10:45:18.857826 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:18.857829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:18.857901 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:18.857995 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:18.858809 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.859729 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.861052 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.861519 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.861629 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:18.867115 (+ 5486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:60351: BlockingRecv error: recv got EOF from 127.8.218.194:60351 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 10:45:18.867362 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:18.856998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:60351)
0928 10:45:18.857568 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:18.857584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:18.857703 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:18.858111 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:18.858116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:18.858517 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:18.858677 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:18.858686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.859893 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.859897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:18.860893 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:18.860907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:18.861687 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:18.861692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:18.861837 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:18.862491 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:18.862511 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:18.866826 (+ 4315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":280,"threads_started":1}
Sep 28 10:45:19 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:19 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:19.233587 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:19.223747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:35933)
0928 10:45:19.224285 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:19.224302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:19.224428 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:19.224766 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:19.224769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:19.225215 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:19.225383 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:19.225393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.226476 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.226483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:19.227564 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:19.227580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.228354 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.228359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:19.228488 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:19.229125 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:19.229143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:19.233202 (+ 4059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":285,"threads_started":1}
W20260928 10:45:19.233728 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:19.223821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:35933 (local address 127.8.218.254:42791)
0928 10:45:19.224018 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:19.224025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:19.224372 (+ 347us) server_negotiation.cc:407] Connection header received
0928 10:45:19.224537 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:19.224541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:19.224637 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:19.224749 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:19.225543 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.226284 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.227724 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.228168 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.228271 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:19.233426 (+ 5155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:35933: BlockingRecv error: recv got EOF from 127.8.218.195:35933 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 10:45:19 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:19 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:19.368358 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:19.358895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:51983)
0928 10:45:19.359416 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:19.359431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:19.359564 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:19.359875 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:19.359879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:19.360329 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:19.360481 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:19.360490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.361524 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.361528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:19.362501 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:19.362514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.363259 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.363265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:19.363465 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:19.364091 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:19.364111 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:19.367847 (+ 3736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":313,"threads_started":1}
W20260928 10:45:19.368417 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:19.358976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:51983 (local address 127.8.218.254:42791)
0928 10:45:19.359174 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:19.359181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:19.359514 (+ 333us) server_negotiation.cc:407] Connection header received
0928 10:45:19.359674 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:19.359679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:19.359742 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:19.359820 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:19.360623 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.361355 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.362630 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.363105 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.363218 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:19.368143 (+ 4925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:51983: BlockingRecv error: recv got EOF from 127.8.218.193:51983 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 10:45:19 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:19 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:19.877676 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:19.868381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:46959)
0928 10:45:19.869152 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:19.869172 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:19.869305 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:19.869695 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:19.869699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:19.870097 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:19.870241 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:19.870249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.871303 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.871308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:19.872284 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:19.872299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.873125 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.873130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:19.873270 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:19.873884 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:19.873902 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:19.877365 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":527,"threads_started":1}
W20260928 10:45:19.877920 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:19.868481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:46959 (local address 127.8.218.254:42791)
0928 10:45:19.868660 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:19.868666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:19.869271 (+ 605us) server_negotiation.cc:407] Connection header received
0928 10:45:19.869398 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:19.869403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:19.869543 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:19.869650 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:19.870356 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.871154 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.872468 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:19.872933 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:19.873037 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:19.877664 (+ 4627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:46959: BlockingRecv error: recv got EOF from 127.8.218.194:46959 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 10:45:20 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:20 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:20.244313 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:20.234847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:43639)
0928 10:45:20.235487 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:20.235512 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:20.235699 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:20.236081 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:20.236084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:20.236517 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:20.236673 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:20.236682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.237771 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.237777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:20.238869 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:20.238884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.239613 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.239618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:20.239766 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:20.240374 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:20.240394 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:20.243976 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":376,"threads_started":1}
W20260928 10:45:20.244580 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:20.234929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:43639 (local address 127.8.218.254:42791)
0928 10:45:20.235163 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:20.235171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:20.235617 (+ 446us) server_negotiation.cc:407] Connection header received
0928 10:45:20.235839 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:20.235845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:20.235938 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:20.236014 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:20.236804 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.237622 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.238998 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.239444 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.239547 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:20.244215 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:43639: BlockingRecv error: recv got EOF from 127.8.218.195:43639 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 10:45:20 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:20 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:20.379423 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:20.369690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:57389)
0928 10:45:20.370162 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:20.370180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:20.370313 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:20.370728 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:20.370732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:20.371108 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:20.371303 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:20.371312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.372482 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.372487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:20.373693 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:20.373708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.374609 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.374616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:20.374783 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:20.375456 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:20.375474 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:20.379059 (+ 3585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
W20260928 10:45:20.379598 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:20.369844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:57389 (local address 127.8.218.254:42791)
0928 10:45:20.370024 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:20.370028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:20.370293 (+ 265us) server_negotiation.cc:407] Connection header received
0928 10:45:20.370410 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:20.370414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:20.370482 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:20.370584 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:20.371492 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.372260 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.373846 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.374407 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.374551 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:20.379312 (+ 4761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:57389: BlockingRecv error: recv got EOF from 127.8.218.193:57389 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 10:45:20 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:20 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:20.890810 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:20.878696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:41065)
0928 10:45:20.879188 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:20.879204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:20.879320 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:20.879598 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:20.879601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:20.879984 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:20.880143 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:20.880152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.881149 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.881155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:20.882171 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:20.882185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.883034 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.883039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:20.883161 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:20.886813 (+ 3652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:20.886833 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:20.890543 (+ 3710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
W20260928 10:45:20.891058 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:20.878827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:41065 (local address 127.8.218.254:42791)
0928 10:45:20.878983 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:20.878987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:20.879274 (+ 287us) server_negotiation.cc:407] Connection header received
0928 10:45:20.879415 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:20.879419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:20.879481 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:20.879568 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:20.880286 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.881015 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.882318 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:20.882853 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:20.882950 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:20.890797 (+ 7847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:41065: BlockingRecv error: recv got EOF from 127.8.218.194:41065 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 10:45:21 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:21 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:21.256119 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:21.245553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:46541)
0928 10:45:21.246185 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:21.246202 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:21.246351 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:21.246737 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:21.246741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:21.247141 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:21.247463 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:21.247477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.248658 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.248663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:21.250150 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:21.250170 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.251367 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.251378 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:21.251520 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:21.252309 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:21.252336 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:21.255862 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":324,"threads_started":1}
W20260928 10:45:21.256477 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:21.245935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:46541 (local address 127.8.218.254:42791)
0928 10:45:21.246182 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:21.246187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:21.246373 (+ 186us) server_negotiation.cc:407] Connection header received
0928 10:45:21.246455 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:21.246459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:21.246535 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:21.246634 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:21.247679 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.248465 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.250397 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.251092 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.251205 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:21.256108 (+ 4903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:46541: BlockingRecv error: recv got EOF from 127.8.218.195:46541 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Sep 28 10:45:21 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:21 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:21.390416 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:21.380616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:53457)
0928 10:45:21.381091 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:21.381108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:21.381231 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:21.381590 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:21.381593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:21.382032 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:21.382241 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:21.382254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.383308 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.383312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:21.384271 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:21.384286 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.385353 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.385358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:21.385535 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:21.386249 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:21.386271 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:21.390150 (+ 3879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":264,"threads_started":1}
W20260928 10:45:21.390674 18328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:21.380684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:53457 (local address 127.8.218.254:42791)
0928 10:45:21.380860 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:21.380865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:21.381216 (+ 351us) server_negotiation.cc:407] Connection header received
0928 10:45:21.381337 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:21.381341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:21.381410 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:21.381543 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:21.382416 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.383170 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.384461 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.384942 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.385076 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:21.390436 (+ 5360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:53457: BlockingRecv error: recv got EOF from 127.8.218.193:53457 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 10:45:21 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:21 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:21.901285 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:21.891829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:44681)
0928 10:45:21.892271 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:21.892288 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:21.892401 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:21.892933 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:21.892939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:21.893377 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:21.893641 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:21.893653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.894823 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.894827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:21.895820 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:21.895840 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.896641 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.896646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:21.896832 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:21.897544 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:21.897566 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:21.901031 (+ 3465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":208,"threads_started":1}
W20260928 10:45:21.901685 18365 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:21.892009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:44681 (local address 127.8.218.254:42791)
0928 10:45:21.892553 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:21.892558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:21.892574 (+ 16us) server_negotiation.cc:407] Connection header received
0928 10:45:21.892622 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:21.892626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:21.892687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:21.892827 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:21.893848 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.894642 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.896013 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:21.896462 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:21.896573 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:21.901344 (+ 4771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:44681: BlockingRecv error: recv got EOF from 127.8.218.194:44681 (error 108)
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":332,"threads_started":1}
Sep 28 10:45:22 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:22 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:22.266575 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:22.257053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:54443)
0928 10:45:22.257612 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:22.257628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:22.257771 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:22.258121 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:22.258127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:22.258569 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:22.258749 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:22.258758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.259772 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.259778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:22.260697 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:22.260711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.261423 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.261429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:22.261647 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:22.262263 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:22.262282 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:22.266258 (+ 3976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":319,"threads_started":1}
W20260928 10:45:22.266741 18365 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:22.257234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:54443 (local address 127.8.218.254:42791)
0928 10:45:22.257501 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:22.257508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:22.257711 (+ 203us) server_negotiation.cc:407] Connection header received
0928 10:45:22.257917 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:22.257921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:22.257984 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:22.258065 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:22.258901 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.259627 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.260852 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.261257 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.261369 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:22.266524 (+ 5155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:54443: BlockingRecv error: recv got EOF from 127.8.218.195:54443 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 28 10:45:22 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:22 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:22.401396 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:22.391532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:48133)
0928 10:45:22.392216 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:22.392235 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:22.392376 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:22.392810 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:22.392813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:22.393393 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:22.393646 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:22.393659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.394888 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.394893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:22.395916 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:22.395931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.396784 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.396792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:22.396948 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:22.397656 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:22.397675 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:22.401113 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":405,"threads_started":1}
W20260928 10:45:22.401835 18365 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:22.391692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:48133 (local address 127.8.218.254:42791)
0928 10:45:22.391915 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:22.391920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:22.392322 (+ 402us) server_negotiation.cc:407] Connection header received
0928 10:45:22.392566 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:22.392571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:22.392665 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:22.392766 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:22.393854 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.394683 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.396095 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.396562 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.396673 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:22.401532 (+ 4859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:48133: BlockingRecv error: recv got EOF from 127.8.218.193:48133 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 10:45:22 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:22 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:22.911708 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:22.902533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:37451)
0928 10:45:22.903016 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:22.903036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:22.903162 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:22.903483 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:22.903486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:22.903888 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:22.904059 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:22.904069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.905150 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.905153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:22.906156 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:22.906172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.906894 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.906898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:22.907042 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:22.907631 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:22.907649 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:22.911447 (+ 3798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":250,"threads_started":1}
W20260928 10:45:22.911890 18369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:22.902540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:37451 (local address 127.8.218.254:42791)
0928 10:45:22.903049 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:22.903056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:22.903095 (+ 39us) server_negotiation.cc:407] Connection header received
0928 10:45:22.903271 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:22.903274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:22.903336 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:22.903443 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:22.904200 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.905022 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.906300 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:22.906759 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:22.906867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:22.911689 (+ 4822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:37451: BlockingRecv error: recv got EOF from 127.8.218.194:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":228,"threads_started":1}
Sep 28 10:45:23 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:23 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:23.278127 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:23.268062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:58237)
0928 10:45:23.268657 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:23.268673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:23.268800 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:23.269187 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:23.269191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:23.269679 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:23.269861 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:23.269871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.271009 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.271013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:23.272115 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:23.272131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.272984 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.272990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:23.273184 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:23.273989 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:23.274009 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:23.277836 (+ 3827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":312,"threads_started":1}
W20260928 10:45:23.278379 18369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:23.268158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:58237 (local address 127.8.218.254:42791)
0928 10:45:23.268364 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:23.268369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:23.268773 (+ 404us) server_negotiation.cc:407] Connection header received
0928 10:45:23.268905 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:23.268911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:23.268995 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:23.269100 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:23.270031 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.270828 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.272262 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.272761 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.272876 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:23.278108 (+ 5232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:58237: BlockingRecv error: recv got EOF from 127.8.218.195:58237 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 10:45:23 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:23 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:23.413226 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:23.402414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:33035)
0928 10:45:23.402968 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:23.402985 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:23.403123 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:23.403525 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:23.403530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:23.403993 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:23.404155 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:23.404166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.405258 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.405262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:23.406282 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:23.406299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.407086 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.407092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:23.407268 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:23.408162 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:23.408182 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:23.412871 (+ 4689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":327,"threads_started":1}
W20260928 10:45:23.413390 18369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:23.402576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:33035 (local address 127.8.218.254:42791)
0928 10:45:23.402781 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:23.402789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:23.403069 (+ 280us) server_negotiation.cc:407] Connection header received
0928 10:45:23.403240 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:23.403245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:23.403328 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:23.403474 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:23.404306 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.405103 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.406478 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.406924 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.407047 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:23.413127 (+ 6080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:33035: BlockingRecv error: recv got EOF from 127.8.218.193:33035 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 10:45:23 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:23 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:23.921818 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:23.912658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:38431)
0928 10:45:23.913109 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:23.913125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:23.913242 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:23.913570 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:23.913572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:23.914026 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:23.914182 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:23.914191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.915155 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.915159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:23.916057 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:23.916071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.916802 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.916805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:23.916926 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:23.917570 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:23.917638 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:23.921424 (+ 3786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
W20260928 10:45:23.922043 18369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:23.912852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:38431 (local address 127.8.218.254:42791)
0928 10:45:23.913075 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:23.913082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:23.913197 (+ 115us) server_negotiation.cc:407] Connection header received
0928 10:45:23.913342 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:23.913346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:23.913427 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:23.913542 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:23.914322 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.915025 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.916214 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:23.916677 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:23.916781 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:23.921823 (+ 5042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:38431: BlockingRecv error: recv got EOF from 127.8.218.194:38431 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 10:45:24 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:24 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.195@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:24.290390 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:24.279260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.195:41809)
0928 10:45:24.279788 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:24.279805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:24.279944 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:24.280463 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:24.280468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:24.281071 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:24.281244 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:24.281254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.282464 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.282469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:24.283549 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:24.283571 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.284641 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.284649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:24.284926 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:24.285796 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:24.285819 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:24.290130 (+ 4311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":303,"threads_started":1}
W20260928 10:45:24.290871 18369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:24.279380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:41809 (local address 127.8.218.254:42791)
0928 10:45:24.279565 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:24.279570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:24.279917 (+ 347us) server_negotiation.cc:407] Connection header received
0928 10:45:24.280158 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:24.280163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:24.280251 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:24.280353 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:24.281480 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.282255 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.283814 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.284296 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.284429 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:24.290616 (+ 6187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.195:41809: BlockingRecv error: recv got EOF from 127.8.218.195:41809 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 10:45:24 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:24 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.193@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:24.425498 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:24.414761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.193:43947)
0928 10:45:24.415339 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:24.415363 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:24.415508 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:24.415862 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:24.415866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:24.416286 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:24.416473 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:24.416483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.417574 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.417581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:24.418570 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:24.418585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.419340 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.419346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:24.419516 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:24.420290 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:24.420313 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:24.425196 (+ 4883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":275,"threads_started":1}
W20260928 10:45:24.425928 18369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:24.414909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:43947 (local address 127.8.218.254:42791)
0928 10:45:24.415237 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:24.415244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:24.415447 (+ 203us) server_negotiation.cc:407] Connection header received
0928 10:45:24.415633 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:24.415645 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:24.415722 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:24.415805 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:24.416630 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.417385 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.418737 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.419183 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.419299 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:24.425684 (+ 6385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.193:43947: BlockingRecv error: recv got EOF from 127.8.218.193:43947 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
Sep 28 10:45:24 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 10:45:24 dist-test-slave-mn38 krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.218.194@KRBTEST.COM for oryx/127.8.218.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 10:45:24.932801 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:24.923308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:42791 (local address 127.8.218.194:42805)
0928 10:45:24.923842 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:24.923860 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:24.924033 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:24.924405 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:24.924411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:24.924809 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:24.924997 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:24.925007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.926316 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.926322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:24.927261 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:24.927275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.928026 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.928032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:24.928209 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:24.928799 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:24.928817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:24.932480 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.218.254:42791: Server oryx/127.8.218.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":294,"threads_started":1}
W20260928 10:45:24.932937 18369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:24.923469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:42805 (local address 127.8.218.254:42791)
0928 10:45:24.923714 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:24.923720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:24.923948 (+ 228us) server_negotiation.cc:407] Connection header received
0928 10:45:24.924187 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:24.924197 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:24.924266 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:24.924349 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:24.925175 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.926157 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.927409 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:24.927859 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:24.927973 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:24.932721 (+ 4748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.218.194:42805: BlockingRecv error: recv got EOF from 127.8.218.194:42805 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:45:25.056031 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 17912
I20260928 10:45:25.069144 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:25.057307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.8.218.193:41565)
0928 10:45:25.057875 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:25.057883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:25.057910 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:45:25.057975 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:25.057981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:25.058221 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:25.058383 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:25.059828 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:25.060895 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:25.062213 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:25.062890 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:25.063008 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:25.065827 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:25.065838 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:25.065855 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:25.065907 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:25.068566 (+ 2659us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.218.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:45:25.068918 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54566: Request ticket server kudu/127.8.218.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
W20260928 10:45:25.069248 18377 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54566: Request ticket server kudu/127.8.218.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:45:25.069351 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 17912: Not authorized: Client connection negotiation failed: client connection to 127.8.218.193:41565: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.218.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:45:25.069561 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 17912
I20260928 10:45:25.084692 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18063
I20260928 10:45:25.098558 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:25.086071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.8.218.194:34129)
0928 10:45:25.086837 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:25.086848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:25.086893 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:45:25.086982 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:25.086991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:25.087376 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:25.087660 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:25.089090 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:25.090372 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:25.091792 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:25.092446 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:25.092570 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:25.095232 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:25.095245 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:25.095261 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:25.095309 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:25.097981 (+ 2672us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.218.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:45:25.098301 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36760: Request ticket server kudu/127.8.218.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":338,"threads_started":1}
W20260928 10:45:25.098662 18378 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36760: Request ticket server kudu/127.8.218.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:45:25.098783 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18063: Not authorized: Client connection negotiation failed: client connection to 127.8.218.194:34129: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.218.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:45:25.098997 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 18063
I20260928 10:45:25.113126 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18196
I20260928 10:45:25.127416 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:25.114443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.8.218.195:34233)
0928 10:45:25.115042 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:25.115050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:25.115078 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:45:25.115153 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:25.115160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:25.115412 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:25.115600 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:25.117370 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:25.118579 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:25.119950 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:25.120612 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:25.120800 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:25.123729 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:25.123740 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:25.123757 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:25.123816 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:25.126803 (+ 2987us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.218.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 10:45:25.127132 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56384: Request ticket server kudu/127.8.218.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":300,"threads_started":1}
W20260928 10:45:25.127527 18379 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56384: Request ticket server kudu/127.8.218.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 10:45:25.127745 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18196: Not authorized: Client connection negotiation failed: client connection to 127.8.218.195:34233: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.218.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 10:45:25.127890 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 18196
I20260928 10:45:25.142268 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 17843
I20260928 10:45:25.143824 17905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:25.262598 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 17843
2026-09-28T10:45:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21438 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T10:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:45:25Z Disabled control of system clock
I20260928 10:45:25.340066 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37831
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:44651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:37831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 10:45:25.687908 18384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:25.688251 18384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:25.688349 18384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:25.696092 18384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:45:25.696221 18384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:25.696261 18384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:25.696338 18384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:45:25.696401 18384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:45:25.706063 18384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:37831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:25.707479 18384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:25.709467 18384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:25.722184 18392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:25.722158 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:25.722223 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:25.723188 18384 server_base.cc:1034] running on GCE node
I20260928 10:45:25.724066 18384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:25.725827 18384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:25.727032 18384 hybrid_clock.cc:648] HybridClock initialized: now 1790592325726983 us; error 47 us; skew 500 ppm
I20260928 10:45:25.727394 18384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:45:25.729817 18384 webserver.cc:460] Webserver started at http://127.8.218.254:40091/ using document root <none> and password file <none>
I20260928 10:45:25.730566 18384 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:25.730664 18384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:25.731012 18384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:25.734176 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "69470957d90f4455b871a851072bba2a"
format_stamp: "Formatted at 2026-09-28 10:45:25 on dist-test-slave-mn38"
server_key: "fa2dc1b8da10c341812bee95921528e2"
server_key_iv: "11cded7e0dc52bc0f0fefc8799efae23"
server_key_version: "encryptionkey@0"
I20260928 10:45:25.735013 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "69470957d90f4455b871a851072bba2a"
format_stamp: "Formatted at 2026-09-28 10:45:25 on dist-test-slave-mn38"
server_key: "fa2dc1b8da10c341812bee95921528e2"
server_key_iv: "11cded7e0dc52bc0f0fefc8799efae23"
server_key_version: "encryptionkey@0"
I20260928 10:45:25.740371 18384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 10:45:25.744252 18398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:25.745951 18384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 10:45:25.746150 18384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "69470957d90f4455b871a851072bba2a"
format_stamp: "Formatted at 2026-09-28 10:45:25 on dist-test-slave-mn38"
server_key: "fa2dc1b8da10c341812bee95921528e2"
server_key_iv: "11cded7e0dc52bc0f0fefc8799efae23"
server_key_version: "encryptionkey@0"
I20260928 10:45:25.746389 18384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:25.771548 18384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:25.772538 18384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:25.772838 18384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:25.791743 18384 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:37831
I20260928 10:45:25.791757 18449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:37831 every 8 connection(s)
I20260928 10:45:25.793256 18384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:45:25.797593 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:25.803629 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 18384
I20260928 10:45:25.803826 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:45:25.804453 9067 external_mini_cluster.cc:1428] Setting key d007eb92f03ae96bab01c4bfb83f02c8
I20260928 10:45:25.809999 18450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a: Bootstrap starting.
I20260928 10:45:25.813730 18450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:25.815073 18450 log.cc:824] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:25.819470 18450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a: No bootstrap required, opened a new log
I20260928 10:45:25.821043 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:25.806834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.8.218.254:37831)
0928 10:45:25.808060 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:25.808074 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:25.808119 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:45:25.809967 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:25.810004 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:25.810671 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:25.811169 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:25.811845 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:25.813205 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:25.814761 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:25.815664 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:25.815952 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:25.817239 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:25.817251 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:25.817274 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:25.818125 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:25.818260 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:25.818305 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:25.818467 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:25.818774 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 10:45:25.819263 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":388,"threads_started":1}
I20260928 10:45:25.829658 18450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69470957d90f4455b871a851072bba2a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37831 } }
I20260928 10:45:25.830076 18450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:25.830152 18450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69470957d90f4455b871a851072bba2a, State: Initialized, Role: FOLLOWER
I20260928 10:45:25.830699 18450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69470957d90f4455b871a851072bba2a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37831 } }
I20260928 10:45:25.830904 18450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:45:25.831007 18450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:45:25.831136 18450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:25.833348 18450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69470957d90f4455b871a851072bba2a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37831 } }
I20260928 10:45:25.833966 18450 leader_election.cc:302] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69470957d90f4455b871a851072bba2a; no voters:
I20260928 10:45:25.834731 18450 leader_election.cc:288] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:45:25.835037 18455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:25.836095 18455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [term 1 LEADER]: Becoming Leader. State: Replica: 69470957d90f4455b871a851072bba2a, State: Running, Role: LEADER
I20260928 10:45:25.836700 18455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69470957d90f4455b871a851072bba2a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37831 } }
I20260928 10:45:25.837189 18450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:45:25.839677 18457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69470957d90f4455b871a851072bba2a. Latest consensus state: current_term: 1 leader_uuid: "69470957d90f4455b871a851072bba2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69470957d90f4455b871a851072bba2a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37831 } } }
I20260928 10:45:25.840159 18457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:25.840747 18456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69470957d90f4455b871a851072bba2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69470957d90f4455b871a851072bba2a" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37831 } } }
I20260928 10:45:25.841099 18456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:25.842528 18463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:45:25.847990 18463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:45:25.859016 18463 catalog_manager.cc:1347] Generated new cluster ID: c4c3e165faed4775a227eb1dca2adacc
I20260928 10:45:25.859143 18463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:45:25.878960 18463 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:45:25.880656 18463 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:45:25.891382 18463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69470957d90f4455b871a851072bba2a: Generated new TSK 0
I20260928 10:45:25.892385 18463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:45:25.901362 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37831
--builtin_ntp_servers=127.8.218.212:44651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 10:45:26.271453 18474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:26.271852 18474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:26.271991 18474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:26.279896 18474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:26.280040 18474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:26.280210 18474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:45:26.289808 18474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:26.291124 18474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:26.292896 18474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:26.306916 18482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:26.307010 18479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:26.307010 18480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:26.307802 18474 server_base.cc:1034] running on GCE node
I20260928 10:45:26.308502 18474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:26.309638 18474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:26.310878 18474 hybrid_clock.cc:648] HybridClock initialized: now 1790592326310801 us; error 70 us; skew 500 ppm
I20260928 10:45:26.311254 18474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:45:26.317746 18474 webserver.cc:460] Webserver started at http://127.8.218.193:38873/ using document root <none> and password file <none>
I20260928 10:45:26.318524 18474 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:26.318624 18474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:26.318964 18474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:26.322435 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "b1a518d5b01c440e8002d90896373b29"
format_stamp: "Formatted at 2026-09-28 10:45:26 on dist-test-slave-mn38"
server_key: "b94bf8a20494ca4dedfc4288337115f9"
server_key_iv: "e811e17c12c046fd3099936d24ac40e5"
server_key_version: "encryptionkey@0"
I20260928 10:45:26.323304 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "b1a518d5b01c440e8002d90896373b29"
format_stamp: "Formatted at 2026-09-28 10:45:26 on dist-test-slave-mn38"
server_key: "b94bf8a20494ca4dedfc4288337115f9"
server_key_iv: "e811e17c12c046fd3099936d24ac40e5"
server_key_version: "encryptionkey@0"
I20260928 10:45:26.329605 18474 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:45:26.333880 18488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:26.336045 18474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:45:26.336345 18474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "b1a518d5b01c440e8002d90896373b29"
format_stamp: "Formatted at 2026-09-28 10:45:26 on dist-test-slave-mn38"
server_key: "b94bf8a20494ca4dedfc4288337115f9"
server_key_iv: "e811e17c12c046fd3099936d24ac40e5"
server_key_version: "encryptionkey@0"
I20260928 10:45:26.336645 18474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:26.359616 18474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:26.360582 18474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:26.360868 18474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:26.361940 18474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:26.363602 18474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:26.363700 18474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:26.363790 18474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:26.363848 18474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:26.397679 18474 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:45577
I20260928 10:45:26.397738 18600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:45577 every 8 connection(s)
I20260928 10:45:26.399154 18474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:45:26.409466 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 18474
I20260928 10:45:26.409684 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:45:26.410334 9067 external_mini_cluster.cc:1428] Setting key 9361d2882ebee067c7d668a2195b3fd3
I20260928 10:45:26.413507 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37831
--builtin_ntp_servers=127.8.218.212:44651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 10:45:26.413921 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:26.402058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:39117 (local address 127.8.218.254:37831)
0928 10:45:26.402569 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:26.402576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:26.404106 (+ 1530us) server_negotiation.cc:407] Connection header received
0928 10:45:26.406156 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:26.406161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:26.406259 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:26.406368 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:26.407767 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:26.408580 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:26.410591 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:26.411042 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:26.411173 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:26.412976 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:26.412981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:26.412985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:26.413005 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:26.413052 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:26.413064 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:26.413140 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:26.413390 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 10:45:26.413553 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":235,"threads_started":1}
I20260928 10:45:26.415098 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:26.402731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37831 (local address 127.8.218.193:39117)
0928 10:45:26.403904 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:26.403953 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:26.405796 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:26.406633 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:26.406644 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:26.406754 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:26.407530 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:26.407551 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:26.408804 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:26.408809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:26.410383 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:26.410405 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:26.411301 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:26.411310 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:26.411667 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:26.412593 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:45:26.412597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:26.412656 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:45:26.412762 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:26.413158 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:26.413170 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:26.413179 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 10:45:26.413367 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 10:45:26.413937 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":324,"threads_started":1}
I20260928 10:45:26.416880 18601 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37831
I20260928 10:45:26.417241 18601 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:26.418161 18601 heartbeater.cc:511] Master 127.8.218.254:37831 requested a full tablet report, sending...
I20260928 10:45:26.420578 18415 ts_manager.cc:194] Registered new tserver with Master: b1a518d5b01c440e8002d90896373b29 (127.8.218.193:45577)
I20260928 10:45:26.422312 18415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.193:39117
W20260928 10:45:26.773051 18605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:26.773357 18605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:26.773492 18605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:26.782060 18605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:26.782198 18605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:26.782342 18605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:45:26.792831 18605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:26.794569 18605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:26.796552 18605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:26.813122 18610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:26.813160 18613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:26.813180 18611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:26.813800 18605 server_base.cc:1034] running on GCE node
I20260928 10:45:26.814484 18605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:26.815559 18605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:26.816780 18605 hybrid_clock.cc:648] HybridClock initialized: now 1790592326816726 us; error 53 us; skew 500 ppm
I20260928 10:45:26.817166 18605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:45:26.819654 18605 webserver.cc:460] Webserver started at http://127.8.218.194:37191/ using document root <none> and password file <none>
I20260928 10:45:26.820427 18605 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:26.820549 18605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:26.820876 18605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:26.824016 18605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "3c6b19f1e4524f23867bcf226b085c19"
format_stamp: "Formatted at 2026-09-28 10:45:26 on dist-test-slave-mn38"
server_key: "cf11fbbb567f00913324070acf94495b"
server_key_iv: "1ddb99a348c81f014c1229fb4b1b226b"
server_key_version: "encryptionkey@0"
I20260928 10:45:26.824898 18605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "3c6b19f1e4524f23867bcf226b085c19"
format_stamp: "Formatted at 2026-09-28 10:45:26 on dist-test-slave-mn38"
server_key: "cf11fbbb567f00913324070acf94495b"
server_key_iv: "1ddb99a348c81f014c1229fb4b1b226b"
server_key_version: "encryptionkey@0"
I20260928 10:45:26.830509 18605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 10:45:26.834152 18619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:26.835760 18605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:45:26.836005 18605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "3c6b19f1e4524f23867bcf226b085c19"
format_stamp: "Formatted at 2026-09-28 10:45:26 on dist-test-slave-mn38"
server_key: "cf11fbbb567f00913324070acf94495b"
server_key_iv: "1ddb99a348c81f014c1229fb4b1b226b"
server_key_version: "encryptionkey@0"
I20260928 10:45:26.836253 18605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:26.851095 18605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:26.852052 18605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:26.852334 18605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:26.853269 18605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:26.854763 18605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:26.854859 18605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:26.854977 18605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:26.855046 18605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:26.886549 18605 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:45945
I20260928 10:45:26.886624 18731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:45945 every 8 connection(s)
I20260928 10:45:26.887940 18605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:45:26.888893 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 18605
I20260928 10:45:26.889081 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:45:26.889654 9067 external_mini_cluster.cc:1428] Setting key e53bd1917c552abb190e2d20e5be6371
I20260928 10:45:26.892763 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37831
--builtin_ntp_servers=127.8.218.212:44651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 10:45:26.902247 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:26.891028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:56895 (local address 127.8.218.254:37831)
0928 10:45:26.891229 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:26.891235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:26.892982 (+ 1747us) server_negotiation.cc:407] Connection header received
0928 10:45:26.894899 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:26.894906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:26.894998 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:26.895083 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:26.896293 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:26.897036 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:26.899082 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:26.899531 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:26.899655 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:26.901363 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:26.901369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:26.901372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:26.901394 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:26.901428 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:26.901467 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:26.901556 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:26.901797 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 10:45:26.901913 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:45:26.903395 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:26.891647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37831 (local address 127.8.218.194:56895)
0928 10:45:26.892802 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:26.892858 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:26.894581 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:26.895349 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:26.895359 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:26.895472 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:26.896102 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:26.896121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:26.897198 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:26.897202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:26.898823 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:26.898842 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:26.899820 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:26.899826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:26.900095 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:26.900979 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:45:26.900982 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:26.901038 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:45:26.901133 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:26.901569 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:26.901582 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:26.901590 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:45:26.901778 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 10:45:26.902241 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":326,"threads_started":1}
I20260928 10:45:26.904973 18732 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37831
I20260928 10:45:26.905294 18732 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:26.906157 18732 heartbeater.cc:511] Master 127.8.218.254:37831 requested a full tablet report, sending...
I20260928 10:45:26.907716 18415 ts_manager.cc:194] Registered new tserver with Master: 3c6b19f1e4524f23867bcf226b085c19 (127.8.218.194:45945)
I20260928 10:45:26.908556 18415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.194:56895
W20260928 10:45:27.235109 18735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:27.235402 18735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:27.235513 18735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:27.243194 18735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:27.243310 18735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:27.243455 18735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:45:27.252908 18735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:27.254298 18735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:27.256093 18735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:27.269402 18740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:27.269517 18743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:27.269483 18741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:27.270358 18735 server_base.cc:1034] running on GCE node
I20260928 10:45:27.271034 18735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:27.272053 18735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:27.273288 18735 hybrid_clock.cc:648] HybridClock initialized: now 1790592327273235 us; error 48 us; skew 500 ppm
I20260928 10:45:27.273676 18735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:45:27.276142 18735 webserver.cc:460] Webserver started at http://127.8.218.195:34437/ using document root <none> and password file <none>
I20260928 10:45:27.276899 18735 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:27.277001 18735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:27.277335 18735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:27.280433 18735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "1d571f9682d54d0e8343dc7cbed68e2f"
format_stamp: "Formatted at 2026-09-28 10:45:27 on dist-test-slave-mn38"
server_key: "9369fa97a07fed457f86882785710a02"
server_key_iv: "8f475fc4483886cbdbcf2e865da33c92"
server_key_version: "encryptionkey@0"
I20260928 10:45:27.281237 18735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "1d571f9682d54d0e8343dc7cbed68e2f"
format_stamp: "Formatted at 2026-09-28 10:45:27 on dist-test-slave-mn38"
server_key: "9369fa97a07fed457f86882785710a02"
server_key_iv: "8f475fc4483886cbdbcf2e865da33c92"
server_key_version: "encryptionkey@0"
I20260928 10:45:27.286842 18735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:45:27.290676 18749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:27.292187 18735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:45:27.292368 18735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "1d571f9682d54d0e8343dc7cbed68e2f"
format_stamp: "Formatted at 2026-09-28 10:45:27 on dist-test-slave-mn38"
server_key: "9369fa97a07fed457f86882785710a02"
server_key_iv: "8f475fc4483886cbdbcf2e865da33c92"
server_key_version: "encryptionkey@0"
I20260928 10:45:27.292591 18735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:27.307727 18735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:27.308620 18735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:27.308898 18735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:27.309875 18735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:27.311314 18735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:27.311408 18735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:27.311520 18735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:27.311589 18735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:27.343437 18735 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:44505
I20260928 10:45:27.343488 18861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:44505 every 8 connection(s)
I20260928 10:45:27.344771 18735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:45:27.346935 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 18735
I20260928 10:45:27.347092 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:45:27.347574 9067 external_mini_cluster.cc:1428] Setting key b943d0bd8a55c76f55aca20daf5b2028
I20260928 10:45:27.358610 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:27.347516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:42045 (local address 127.8.218.254:37831)
0928 10:45:27.347726 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:27.347732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:27.349702 (+ 1970us) server_negotiation.cc:407] Connection header received
0928 10:45:27.351862 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:27.351866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:27.351938 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:27.352034 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:27.353245 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.354132 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.355809 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.356214 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.356316 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:27.357814 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:27.357819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:27.357822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:27.357841 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:27.357872 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:27.357884 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:27.357964 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:27.358248 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 10:45:27.358333 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:45:27.359602 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:27.348115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37831 (local address 127.8.218.195:42045)
0928 10:45:27.349545 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:27.349589 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:27.351566 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:27.352234 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:27.352244 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:27.352356 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:27.353068 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:27.353088 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.354317 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.354321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:27.355646 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:27.355659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.356454 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.356459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:27.356680 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:27.357422 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:45:27.357424 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:27.357520 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:45:27.357612 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:27.357978 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:27.357987 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:27.357994 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:45:27.358159 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 10:45:27.358526 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":517,"threads_started":1}
I20260928 10:45:27.361120 18862 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37831
I20260928 10:45:27.361522 18862 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:27.362520 18862 heartbeater.cc:511] Master 127.8.218.254:37831 requested a full tablet report, sending...
I20260928 10:45:27.364238 18415 ts_manager.cc:194] Registered new tserver with Master: 1d571f9682d54d0e8343dc7cbed68e2f (127.8.218.195:44505)
I20260928 10:45:27.365034 18415 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.195:42045
I20260928 10:45:27.375808 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 10:45:27.381881 18603 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:27.380384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.8.218.254:37831)
0928 10:45:27.380684 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:27.380690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:27.380941 (+ 251us) server_negotiation.cc:407] Connection header received
0928 10:45:27.381100 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:27.381105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:27.381174 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:27.381248 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:27.381655 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60186: BlockingRecv error: recv got EOF from 127.0.0.1:60186 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 10:45:27.382539 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18474
I20260928 10:45:27.390432 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:27.383405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.8.218.193:45577)
0928 10:45:27.383919 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:27.383926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:27.383951 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:45:27.384005 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:27.384012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:27.384234 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:27.384357 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:27.384705 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.385751 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.387143 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.387725 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.387823 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:27.388937 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:27.388946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:27.388960 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:27.389684 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:27.389734 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:27.389754 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:27.389843 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:27.390025 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 10:45:27.390141 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":269,"threads_started":1}
I20260928 10:45:27.391746 18596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:27.474457 18601 heartbeater.cc:503] Master 127.8.218.254:37831 was elected leader, sending a full tablet report...
I20260928 10:45:27.475497 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 18474
W20260928 10:45:27.489003 18400 connection.cc:535] server connection from 127.8.218.193:39117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 10:45:27.489256 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18605
I20260928 10:45:27.497066 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:27.490165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.8.218.194:45945)
0928 10:45:27.490694 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:27.490701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:27.490726 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:45:27.490785 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:27.490791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:27.491023 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:27.491176 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:27.491554 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.492535 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.493847 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.494408 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.494541 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:27.495583 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:27.495591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:27.495606 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:27.496206 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:27.496255 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:27.496275 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:27.496397 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:27.496555 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 10:45:27.496736 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":236,"threads_started":1}
I20260928 10:45:27.498356 18727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:27.580657 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 18605
I20260928 10:45:27.595309 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18735
I20260928 10:45:27.604924 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:27.596396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.8.218.195:44505)
0928 10:45:27.597046 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:27.597055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:27.597095 (+ 40us) server_negotiation.cc:407] Connection header received
0928 10:45:27.597161 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:27.597167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:27.597566 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:27.597734 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:27.598310 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.599493 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.600923 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:27.601928 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:27.602158 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:27.603350 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:27.603360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:27.603376 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:27.604020 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:27.604086 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:27.604115 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:27.604213 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:27.604376 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 10:45:27.604519 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":348,"threads_started":1}
I20260928 10:45:27.606601 18857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:27.688886 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 18735
I20260928 10:45:27.702297 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18384
I20260928 10:45:27.703500 18445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:27.806020 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 18384
2026-09-28T10:45:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2537 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T10:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:45:27Z Disabled control of system clock
I20260928 10:45:27.874538 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:32823
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 10:45:28.207476 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:28.207801 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:28.207890 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:28.215277 18881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:45:28.215407 18881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:28.215443 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:28.215507 18881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:45:28.215549 18881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:45:28.225497 18881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:32823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:28.226974 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:28.228812 18881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:28.240733 18886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:28.240758 18887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:28.240958 18889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:28.242108 18881 server_base.cc:1034] running on GCE node
I20260928 10:45:28.243013 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:28.244679 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:28.245918 18881 hybrid_clock.cc:648] HybridClock initialized: now 1790592328245864 us; error 45 us; skew 500 ppm
I20260928 10:45:28.246400 18881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:45:28.248843 18881 webserver.cc:460] Webserver started at http://127.8.218.254:46079/ using document root <none> and password file <none>
I20260928 10:45:28.249650 18881 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:28.249749 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:28.250067 18881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:28.253109 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "60d65708c7974c24a2b56777caaaad26"
format_stamp: "Formatted at 2026-09-28 10:45:28 on dist-test-slave-mn38"
server_key: "c6d5d9f499f85971696f8c50e28c8725"
server_key_iv: "9cf04e176861766f5d375cd1c4e1990e"
server_key_version: "encryptionkey@0"
I20260928 10:45:28.254021 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "60d65708c7974c24a2b56777caaaad26"
format_stamp: "Formatted at 2026-09-28 10:45:28 on dist-test-slave-mn38"
server_key: "c6d5d9f499f85971696f8c50e28c8725"
server_key_iv: "9cf04e176861766f5d375cd1c4e1990e"
server_key_version: "encryptionkey@0"
I20260928 10:45:28.259778 18881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:45:28.263774 18895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:28.265347 18881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:45:28.265584 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "60d65708c7974c24a2b56777caaaad26"
format_stamp: "Formatted at 2026-09-28 10:45:28 on dist-test-slave-mn38"
server_key: "c6d5d9f499f85971696f8c50e28c8725"
server_key_iv: "9cf04e176861766f5d375cd1c4e1990e"
server_key_version: "encryptionkey@0"
I20260928 10:45:28.265818 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:28.272073 18881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:28.272977 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:28.273288 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:28.290829 18881 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:32823
I20260928 10:45:28.290843 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:32823 every 8 connection(s)
I20260928 10:45:28.292217 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:45:28.296237 18947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:28.297648 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 18881
I20260928 10:45:28.297822 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:45:28.298520 9067 external_mini_cluster.cc:1428] Setting key ecfff3deb3d2735b4345a67ac8a6ad0f
I20260928 10:45:28.307657 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:28.300555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.8.218.254:32823)
0928 10:45:28.301973 (+ 1418us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:28.301990 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:28.302063 (+ 73us) server_negotiation.cc:407] Connection header received
0928 10:45:28.303876 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:28.303903 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:28.304362 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:28.304752 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:28.305889 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:28.305896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:28.305911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:28.305929 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:28.305996 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:28.306022 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:28.306260 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:28.306421 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 10:45:28.306685 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":469,"threads_started":1}
I20260928 10:45:28.308710 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26: Bootstrap starting.
I20260928 10:45:28.312305 18947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:28.313427 18947 log.cc:824] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:28.316604 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26: No bootstrap required, opened a new log
I20260928 10:45:28.324620 18947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d65708c7974c24a2b56777caaaad26" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32823 } }
I20260928 10:45:28.325029 18947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:28.325103 18947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d65708c7974c24a2b56777caaaad26, State: Initialized, Role: FOLLOWER
I20260928 10:45:28.325798 18947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d65708c7974c24a2b56777caaaad26" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32823 } }
I20260928 10:45:28.326017 18947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:45:28.326138 18947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:45:28.326316 18947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:28.328709 18947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d65708c7974c24a2b56777caaaad26" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32823 } }
I20260928 10:45:28.329372 18947 leader_election.cc:302] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60d65708c7974c24a2b56777caaaad26; no voters:
I20260928 10:45:28.330096 18947 leader_election.cc:288] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:45:28.330235 18952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:28.330920 18952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [term 1 LEADER]: Becoming Leader. State: Replica: 60d65708c7974c24a2b56777caaaad26, State: Running, Role: LEADER
I20260928 10:45:28.331508 18952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d65708c7974c24a2b56777caaaad26" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32823 } }
I20260928 10:45:28.332144 18947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:45:28.334199 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60d65708c7974c24a2b56777caaaad26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d65708c7974c24a2b56777caaaad26" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32823 } } }
I20260928 10:45:28.334585 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:28.335835 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60d65708c7974c24a2b56777caaaad26. Latest consensus state: current_term: 1 leader_uuid: "60d65708c7974c24a2b56777caaaad26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d65708c7974c24a2b56777caaaad26" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32823 } } }
I20260928 10:45:28.336238 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26 [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:28.336426 18958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:45:28.341418 18958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:45:28.352106 18958 catalog_manager.cc:1347] Generated new cluster ID: 02e5e1cd009d4f81aeb1ad051a19f631
I20260928 10:45:28.352231 18958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:45:28.365418 18958 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:45:28.366001 18958 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:45:28.377136 18958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60d65708c7974c24a2b56777caaaad26: Generated new TSK 0
I20260928 10:45:28.378090 18958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:45:28.383481 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32823
--builtin_ntp_servers=127.8.218.212:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 10:45:28.744735 18971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:28.745038 18971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:28.745174 18971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:28.752663 18971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:28.752776 18971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:28.752923 18971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:45:28.762490 18971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:28.763819 18971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:28.765745 18971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:28.779950 18976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:28.780020 18979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:28.779997 18977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:28.780871 18971 server_base.cc:1034] running on GCE node
I20260928 10:45:28.781626 18971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:28.782711 18971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:28.783931 18971 hybrid_clock.cc:648] HybridClock initialized: now 1790592328783871 us; error 56 us; skew 500 ppm
I20260928 10:45:28.784318 18971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:45:28.786723 18971 webserver.cc:460] Webserver started at http://127.8.218.193:43531/ using document root <none> and password file <none>
I20260928 10:45:28.787657 18971 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:28.787777 18971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:28.788152 18971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:28.791751 18971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "25c7cbe15b214e7fad14a53bff6d62f0"
format_stamp: "Formatted at 2026-09-28 10:45:28 on dist-test-slave-mn38"
server_key: "64c3b01e24668c2841be5a27966136a6"
server_key_iv: "43317b4b39e583a7793cb3efa7d9b7a2"
server_key_version: "encryptionkey@0"
I20260928 10:45:28.792872 18971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "25c7cbe15b214e7fad14a53bff6d62f0"
format_stamp: "Formatted at 2026-09-28 10:45:28 on dist-test-slave-mn38"
server_key: "64c3b01e24668c2841be5a27966136a6"
server_key_iv: "43317b4b39e583a7793cb3efa7d9b7a2"
server_key_version: "encryptionkey@0"
I20260928 10:45:28.799348 18971 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260928 10:45:28.803409 18985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:28.805120 18971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:45:28.805318 18971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "25c7cbe15b214e7fad14a53bff6d62f0"
format_stamp: "Formatted at 2026-09-28 10:45:28 on dist-test-slave-mn38"
server_key: "64c3b01e24668c2841be5a27966136a6"
server_key_iv: "43317b4b39e583a7793cb3efa7d9b7a2"
server_key_version: "encryptionkey@0"
I20260928 10:45:28.805594 18971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:28.811728 18971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:28.812626 18971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:28.812899 18971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:28.813974 18971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:28.815579 18971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:28.815686 18971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:28.815795 18971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:28.815861 18971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:28.848389 18971 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:43201
I20260928 10:45:28.848443 19097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:43201 every 8 connection(s)
I20260928 10:45:28.849727 18971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:45:28.850726 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 18971
I20260928 10:45:28.850914 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:45:28.851742 9067 external_mini_cluster.cc:1428] Setting key 4ee99a340e4ca6026b94700dbc4b1c8c
I20260928 10:45:28.854992 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32823
--builtin_ntp_servers=127.8.218.212:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 10:45:28.859654 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:28.852661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:38663 (local address 127.8.218.254:32823)
0928 10:45:28.853127 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:28.853131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:28.854621 (+ 1490us) server_negotiation.cc:407] Connection header received
0928 10:45:28.856947 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:28.856951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:28.857013 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:28.857106 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:28.858617 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:28.858621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:28.858624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:28.858640 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:28.858685 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:28.858698 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:28.858801 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:28.859016 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 10:45:28.859117 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":219,"threads_started":1}
I20260928 10:45:28.860587 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:28.853320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:32823 (local address 127.8.218.193:38663)
0928 10:45:28.854449 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:28.854501 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:28.856632 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:28.857428 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:28.857466 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:28.857595 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:28.858269 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:45:28.858273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:28.858333 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:45:28.858441 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:28.858784 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:28.858793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:28.858800 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:45:28.859054 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 10:45:28.859323 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":320,"threads_started":1}
I20260928 10:45:28.862119 19098 heartbeater.cc:347] Connected to a master server at 127.8.218.254:32823
I20260928 10:45:28.862438 19098 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:28.863152 19098 heartbeater.cc:511] Master 127.8.218.254:32823 requested a full tablet report, sending...
I20260928 10:45:28.865561 18912 ts_manager.cc:194] Registered new tserver with Master: 25c7cbe15b214e7fad14a53bff6d62f0 (127.8.218.193:43201)
W20260928 10:45:29.184437 19102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:29.184718 19102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:29.184805 19102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:29.192106 19102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:29.192216 19102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:29.192327 19102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:45:29.201460 19102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:29.202706 19102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:29.204460 19102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:29.217582 19110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:29.217638 19107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:29.217618 19108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:29.218179 19102 server_base.cc:1034] running on GCE node
I20260928 10:45:29.218837 19102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:29.219843 19102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:29.221076 19102 hybrid_clock.cc:648] HybridClock initialized: now 1790592329220992 us; error 69 us; skew 500 ppm
I20260928 10:45:29.221474 19102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:45:29.223910 19102 webserver.cc:460] Webserver started at http://127.8.218.194:33089/ using document root <none> and password file <none>
I20260928 10:45:29.224666 19102 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:29.224786 19102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:29.225106 19102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:29.228181 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "9624ebc4081842d4b2201d55cd3853a5"
format_stamp: "Formatted at 2026-09-28 10:45:29 on dist-test-slave-mn38"
server_key: "48fa413b2ccb110bcb5c6ac101292e69"
server_key_iv: "7e2855df78e75708097653a3c6dc236e"
server_key_version: "encryptionkey@0"
I20260928 10:45:29.229039 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "9624ebc4081842d4b2201d55cd3853a5"
format_stamp: "Formatted at 2026-09-28 10:45:29 on dist-test-slave-mn38"
server_key: "48fa413b2ccb110bcb5c6ac101292e69"
server_key_iv: "7e2855df78e75708097653a3c6dc236e"
server_key_version: "encryptionkey@0"
I20260928 10:45:29.234726 19102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 10:45:29.238479 19116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:29.240234 19102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:45:29.240437 19102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "9624ebc4081842d4b2201d55cd3853a5"
format_stamp: "Formatted at 2026-09-28 10:45:29 on dist-test-slave-mn38"
server_key: "48fa413b2ccb110bcb5c6ac101292e69"
server_key_iv: "7e2855df78e75708097653a3c6dc236e"
server_key_version: "encryptionkey@0"
I20260928 10:45:29.240662 19102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:29.247099 19102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:29.248093 19102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:29.248376 19102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:29.249411 19102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:29.251101 19102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:29.251220 19102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:29.251338 19102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:29.251406 19102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:29.283377 19102 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:46623
I20260928 10:45:29.283424 19228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:46623 every 8 connection(s)
I20260928 10:45:29.284726 19102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:45:29.291087 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 19102
I20260928 10:45:29.291255 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:45:29.291782 9067 external_mini_cluster.cc:1428] Setting key 62d06b1106e13b21e17640eb2b030443
I20260928 10:45:29.293336 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:29.287387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:43595 (local address 127.8.218.254:32823)
0928 10:45:29.287587 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:29.287592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:29.289035 (+ 1443us) server_negotiation.cc:407] Connection header received
0928 10:45:29.290878 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:29.290884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:29.290971 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:29.291064 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:29.292527 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:29.292533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:29.292539 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:29.292562 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:29.292603 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:29.292617 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:29.292737 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:29.292985 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 10:45:29.293051 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:45:29.294533 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:29.287908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:32823 (local address 127.8.218.194:43595)
0928 10:45:29.288863 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:29.288906 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:29.290596 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:29.291367 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:29.291375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:29.291480 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:29.292163 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:45:29.292165 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:29.292206 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:45:29.292291 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:29.292754 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:29.292765 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:29.292773 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:45:29.293055 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 10:45:29.293343 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":284,"threads_started":1}
I20260928 10:45:29.295019 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32823
--builtin_ntp_servers=127.8.218.212:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 10:45:29.296087 19229 heartbeater.cc:347] Connected to a master server at 127.8.218.254:32823
I20260928 10:45:29.296417 19229 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:29.297168 19229 heartbeater.cc:511] Master 127.8.218.254:32823 requested a full tablet report, sending...
I20260928 10:45:29.298934 18912 ts_manager.cc:194] Registered new tserver with Master: 9624ebc4081842d4b2201d55cd3853a5 (127.8.218.194:46623)
W20260928 10:45:29.635264 19232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:29.635583 19232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:29.635699 19232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:29.643349 19232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:29.643477 19232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:29.643627 19232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:45:29.653173 19232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:29.654546 19232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:29.656347 19232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:29.669214 19240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:29.669174 19238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:29.669289 19237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:29.670423 19232 server_base.cc:1034] running on GCE node
I20260928 10:45:29.671065 19232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:29.672200 19232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:29.673412 19232 hybrid_clock.cc:648] HybridClock initialized: now 1790592329673362 us; error 46 us; skew 500 ppm
I20260928 10:45:29.673806 19232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:45:29.676354 19232 webserver.cc:460] Webserver started at http://127.8.218.195:38239/ using document root <none> and password file <none>
I20260928 10:45:29.677079 19232 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:29.677181 19232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:29.677567 19232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:29.680506 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "ae1419534a934467bf2d8335ad922340"
format_stamp: "Formatted at 2026-09-28 10:45:29 on dist-test-slave-mn38"
server_key: "bcb95852bc5a3ce48bf2fa6f940bb65d"
server_key_iv: "ecf704cdaec4bb34ec946c0e8f6a7437"
server_key_version: "encryptionkey@0"
I20260928 10:45:29.681267 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "ae1419534a934467bf2d8335ad922340"
format_stamp: "Formatted at 2026-09-28 10:45:29 on dist-test-slave-mn38"
server_key: "bcb95852bc5a3ce48bf2fa6f940bb65d"
server_key_iv: "ecf704cdaec4bb34ec946c0e8f6a7437"
server_key_version: "encryptionkey@0"
I20260928 10:45:29.686439 19232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:45:29.690127 19246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:29.691633 19232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:45:29.691818 19232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "ae1419534a934467bf2d8335ad922340"
format_stamp: "Formatted at 2026-09-28 10:45:29 on dist-test-slave-mn38"
server_key: "bcb95852bc5a3ce48bf2fa6f940bb65d"
server_key_iv: "ecf704cdaec4bb34ec946c0e8f6a7437"
server_key_version: "encryptionkey@0"
I20260928 10:45:29.692049 19232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:29.698101 19232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:29.699005 19232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:29.699285 19232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:29.700310 19232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:29.702153 19232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:29.702396 19232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:29.702543 19232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:29.702735 19232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:29.734413 19232 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:46305
I20260928 10:45:29.734477 19358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:46305 every 8 connection(s)
I20260928 10:45:29.735702 19232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:45:29.739003 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 19232
I20260928 10:45:29.739151 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:45:29.739609 9067 external_mini_cluster.cc:1428] Setting key 96937278967016cea1d8d045be219c77
I20260928 10:45:29.744233 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:29.738260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:35399 (local address 127.8.218.254:32823)
0928 10:45:29.738423 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:29.738427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:29.740057 (+ 1630us) server_negotiation.cc:407] Connection header received
0928 10:45:29.741960 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:29.741966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:29.742046 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:29.742148 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:29.743608 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:29.743631 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:29.743635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:29.743652 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:29.743693 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:29.743706 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:29.743822 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:29.743968 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 10:45:29.744017 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 10:45:29.745261 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:29.738770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:32823 (local address 127.8.218.195:35399)
0928 10:45:29.739872 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:29.739916 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:29.741668 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:29.742460 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:29.742470 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:29.742575 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:29.743266 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:45:29.743270 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:29.743329 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:45:29.743427 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:29.743804 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:29.743811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:29.743817 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:45:29.744031 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 10:45:29.744250 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":305,"threads_started":1}
I20260928 10:45:29.746825 19359 heartbeater.cc:347] Connected to a master server at 127.8.218.254:32823
I20260928 10:45:29.747117 19359 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:29.747763 19359 heartbeater.cc:511] Master 127.8.218.254:32823 requested a full tablet report, sending...
I20260928 10:45:29.749251 18912 ts_manager.cc:194] Registered new tserver with Master: ae1419534a934467bf2d8335ad922340 (127.8.218.195:46305)
I20260928 10:45:29.755343 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 10:45:29.761812 19100 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:45:29.759953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.8.218.254:32823)
0928 10:45:29.760162 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:29.760167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:29.760696 (+ 529us) server_negotiation.cc:407] Connection header received
0928 10:45:29.760894 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:29.760899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:29.760957 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:29.761047 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:29.761540 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51704: BlockingRecv error: recv got EOF from 127.0.0.1:51704 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:45:29.762459 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18971
I20260928 10:45:29.766105 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:29.763515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.8.218.193:43201)
0928 10:45:29.764045 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:29.764052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:29.764079 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:45:29.764136 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:29.764143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:29.764369 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:29.764497 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:29.765366 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:29.765372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:29.765384 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:29.765401 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:29.765541 (+ 140us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:29.765576 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:29.765725 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:29.765794 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 10:45:29.765874 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":286,"threads_started":1}
I20260928 10:45:29.767344 19093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:29.848898 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 18971
I20260928 10:45:29.862058 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 19102
I20260928 10:45:29.866163 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:29.863109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.8.218.194:46623)
0928 10:45:29.863704 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:29.863715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:29.863749 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:45:29.863826 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:29.863834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:29.864131 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:29.864439 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:29.865348 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:29.865356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:29.865368 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:29.865383 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:29.865460 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:29.865484 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:29.865607 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:29.865755 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 10:45:29.865857 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":273,"threads_started":1}
I20260928 10:45:29.867472 19224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:29.953323 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 19102
I20260928 10:45:29.966599 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 19232
I20260928 10:45:29.971019 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:29.967676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.8.218.195:46305)
0928 10:45:29.968440 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:29.968447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:29.968480 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:45:29.968550 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:29.968556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:29.968784 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:29.968913 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:29.970060 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:29.970067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:29.970080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:45:29.970104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:29.970172 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:45:29.970193 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:29.970329 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:29.970522 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 10:45:29.970657 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":378,"threads_started":1}
I20260928 10:45:29.972317 19354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:30.054106 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 19232
I20260928 10:45:30.067147 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 18881
I20260928 10:45:30.068445 18942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:30.168671 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 18881
2026-09-28T10:45:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2362 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:45:30 dist-test-slave-mn38 krb5kdc[19377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:45:30 dist-test-slave-mn38 krb5kdc[19377](info): set up 2 sockets
Sep 28 10:45:30 dist-test-slave-mn38 krb5kdc[19377](info): commencing operation
krb5kdc: starting...
W20260928 10:45:31.223397 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:45:31 dist-test-slave-mn38 krb5kdc[19377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:45:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:45:31.461864 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36423
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:31.797384 19390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:31.797761 19390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:31.797891 19390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:31.805467 19390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:45:31.805600 19390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:31.805660 19390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:31.805733 19390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:45:31.805779 19390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:45:31.816228 19390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:31.817656 19390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:31.819005 19390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:31.831050 19395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:31.831056 19396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:31.831053 19398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:31.831888 19390 server_base.cc:1034] running on GCE node
I20260928 10:45:31.832765 19390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:31.834417 19390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:31.835649 19390 hybrid_clock.cc:648] HybridClock initialized: now 1790592331835600 us; error 41 us; skew 500 ppm
I20260928 10:45:31.836024 19390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:31 dist-test-slave-mn38 krb5kdc[19377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592331, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:31.842844 19390 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:45:31.844587 19390 webserver.cc:460] Webserver started at http://127.8.218.254:36693/ using document root <none> and password file <none>
I20260928 10:45:31.845331 19390 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:31.845479 19390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:31.845799 19390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:31.848944 19390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "bc7bfa8cefce463ead143770f5156127"
format_stamp: "Formatted at 2026-09-28 10:45:31 on dist-test-slave-mn38"
server_key: "43b02ae96e2698b54a6b32f355ac57f7"
server_key_iv: "a5dcd6b2be7904f7217a0e99f0f1c58e"
server_key_version: "encryptionkey@0"
I20260928 10:45:31.849761 19390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "bc7bfa8cefce463ead143770f5156127"
format_stamp: "Formatted at 2026-09-28 10:45:31 on dist-test-slave-mn38"
server_key: "43b02ae96e2698b54a6b32f355ac57f7"
server_key_iv: "a5dcd6b2be7904f7217a0e99f0f1c58e"
server_key_version: "encryptionkey@0"
I20260928 10:45:31.855170 19390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 10:45:31.858879 19405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:31.860323 19390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:45:31.860539 19390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "bc7bfa8cefce463ead143770f5156127"
format_stamp: "Formatted at 2026-09-28 10:45:31 on dist-test-slave-mn38"
server_key: "43b02ae96e2698b54a6b32f355ac57f7"
server_key_iv: "a5dcd6b2be7904f7217a0e99f0f1c58e"
server_key_version: "encryptionkey@0"
I20260928 10:45:31.860735 19390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:31.889714 19390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:31.894199 19390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:31.894480 19390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:31.912954 19390 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:36423
I20260928 10:45:31.912972 19456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:36423 every 8 connection(s)
I20260928 10:45:31.914413 19390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:45:31.915274 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 19390
I20260928 10:45:31.915503 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:45:31.916131 9067 external_mini_cluster.cc:1428] Setting key 699a00c3440cb29f604118d97f867ddd
I20260928 10:45:31.919085 19457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:45:31 dist-test-slave-mn38 krb5kdc[19377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:45:31.932408 19457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127: Bootstrap starting.
I20260928 10:45:31.935894 19457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:31.937083 19457 log.cc:824] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:31.940516 19457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127: No bootstrap required, opened a new log
I20260928 10:45:31.940546 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:31.918303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.8.218.254:36423)
0928 10:45:31.919149 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:31.919168 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:31.919223 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:45:31.921494 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:31.921533 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:31.922074 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:31.922588 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:31.923920 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:31.926044 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:31.927442 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:31.928498 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:31.928680 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:31.932799 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:31.932808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:31.932825 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:31.932873 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:31.935957 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:31.936731 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:31.936742 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:31.936751 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:31.936995 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:31.937337 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:31.937342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:31.937345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:31.938042 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:31.938278 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:31.938784 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 10:45:31.939227 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":350,"threads_started":1}
I20260928 10:45:31.947839 19457 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7bfa8cefce463ead143770f5156127" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36423 } }
I20260928 10:45:31.948207 19457 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:31.948287 19457 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7bfa8cefce463ead143770f5156127, State: Initialized, Role: FOLLOWER
I20260928 10:45:31.948807 19457 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7bfa8cefce463ead143770f5156127" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36423 } }
I20260928 10:45:31.949044 19457 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:45:31.949137 19457 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:45:31.949265 19457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:31.951403 19457 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7bfa8cefce463ead143770f5156127" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36423 } }
I20260928 10:45:31.951869 19457 leader_election.cc:302] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7bfa8cefce463ead143770f5156127; no voters:
I20260928 10:45:31.952476 19457 leader_election.cc:288] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:45:31.952594 19462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:31.953214 19462 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [term 1 LEADER]: Becoming Leader. State: Replica: bc7bfa8cefce463ead143770f5156127, State: Running, Role: LEADER
I20260928 10:45:31.953822 19462 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7bfa8cefce463ead143770f5156127" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36423 } }
I20260928 10:45:31.954696 19457 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:45:31.956329 19464 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc7bfa8cefce463ead143770f5156127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7bfa8cefce463ead143770f5156127" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36423 } } }
I20260928 10:45:31.956707 19464 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:31.957497 19463 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7bfa8cefce463ead143770f5156127. Latest consensus state: current_term: 1 leader_uuid: "bc7bfa8cefce463ead143770f5156127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7bfa8cefce463ead143770f5156127" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36423 } } }
I20260928 10:45:31.957891 19463 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127 [sys.catalog]: This master's current role is: LEADER
I20260928 10:45:31.961020 19467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:45:31.965679 19467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:45:31.974779 19467 catalog_manager.cc:1347] Generated new cluster ID: e60fc37cc0be4670b46f96882d12e45f
I20260928 10:45:31.974910 19467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:45:31.992120 19467 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:45:31.993418 19467 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:45:32.002103 19467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127: Generated new TSK 0
I20260928 10:45:32.002972 19467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:45:32.065056 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36423
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:32.405100 19483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:32.405406 19483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:32.405576 19483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:32.413225 19483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:32.413362 19483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:32.413537 19483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:45:32.423736 19483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:32.425120 19483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:32.426563 19483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:32.439934 19488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:32.440127 19491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:32.440488 19489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:32.440781 19483 server_base.cc:1034] running on GCE node
I20260928 10:45:32.441356 19483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:32.442327 19483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:32.443519 19483 hybrid_clock.cc:648] HybridClock initialized: now 1790592332443478 us; error 40 us; skew 500 ppm
I20260928 10:45:32.443809 19483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:32 dist-test-slave-mn38 krb5kdc[19377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592332, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:32.446985 19483 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:45:32.448431 19483 webserver.cc:460] Webserver started at http://127.8.218.193:34591/ using document root <none> and password file <none>
I20260928 10:45:32.449084 19483 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:32.449182 19483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:32.449462 19483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:32.452409 19483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "37041af7ddbc4b7d8888b03c8a1a17ce"
format_stamp: "Formatted at 2026-09-28 10:45:32 on dist-test-slave-mn38"
server_key: "529157ac6f775c6130dbbc2edbddde93"
server_key_iv: "3928ce422111e43489f4f07dc1608313"
server_key_version: "encryptionkey@0"
I20260928 10:45:32.453153 19483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "37041af7ddbc4b7d8888b03c8a1a17ce"
format_stamp: "Formatted at 2026-09-28 10:45:32 on dist-test-slave-mn38"
server_key: "529157ac6f775c6130dbbc2edbddde93"
server_key_iv: "3928ce422111e43489f4f07dc1608313"
server_key_version: "encryptionkey@0"
I20260928 10:45:32.458537 19483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 10:45:32.462319 19498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:32.463820 19483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:45:32.463964 19483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "37041af7ddbc4b7d8888b03c8a1a17ce"
format_stamp: "Formatted at 2026-09-28 10:45:32 on dist-test-slave-mn38"
server_key: "529157ac6f775c6130dbbc2edbddde93"
server_key_iv: "3928ce422111e43489f4f07dc1608313"
server_key_version: "encryptionkey@0"
I20260928 10:45:32.464144 19483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:32.480098 19483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:32.484665 19483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:32.484964 19483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:32.486045 19483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:32.487493 19483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:32.487589 19483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:32.487687 19483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:32.487748 19483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:32.519343 19483 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:38653
I20260928 10:45:32.519356 19610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:38653 every 8 connection(s)
I20260928 10:45:32.520665 19483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:45:32.528702 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 19483
I20260928 10:45:32.528896 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:45:32.529567 9067 external_mini_cluster.cc:1428] Setting key 78bb7d86455d764b1af19604f1f7f4b9
Sep 28 10:45:32 dist-test-slave-mn38 krb5kdc[19377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592332, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:45:32.542629 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:32.523294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:38727 (local address 127.8.218.254:36423)
0928 10:45:32.523807 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:32.523812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:32.524742 (+ 930us) server_negotiation.cc:407] Connection header received
0928 10:45:32.526652 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:32.526658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:32.526746 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:32.526844 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:32.528558 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:32.529320 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:32.531345 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:32.531861 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:32.532014 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:32.535947 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:32.535953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:32.535958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:32.536001 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:32.539065 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:32.540068 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:32.540073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:32.540075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:32.540198 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:32.540792 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:32.540796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:32.540798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:32.541136 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:32.541257 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:32.542162 (+ 905us) server_negotiation.cc:299] Negotiation successful
0928 10:45:32.542345 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":321,"threads_started":1}
I20260928 10:45:32.543969 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:32.523814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36423 (local address 127.8.218.193:38727)
0928 10:45:32.524569 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:32.524615 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:32.526344 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:32.527123 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:32.527132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:32.527707 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:32.528344 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:32.528380 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:32.529517 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:32.529522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:32.531149 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:32.531165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:32.532140 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:32.532146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:32.532356 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:32.533148 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:32.533176 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:32.535651 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:32.539267 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:32.539276 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:32.539293 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:32.539925 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:32.540307 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:32.540312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:32.540316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:32.540642 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:32.541263 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:32.541271 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:32.541755 (+ 484us) client_negotiation.cc:769] Sending connection context
0928 10:45:32.542032 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 10:45:32.542621 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":286,"threads_started":1}
I20260928 10:45:32.545729 19611 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36423
I20260928 10:45:32.546072 19611 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:32.547515 19611 heartbeater.cc:511] Master 127.8.218.254:36423 requested a full tablet report, sending...
I20260928 10:45:32.550086 19422 ts_manager.cc:194] Registered new tserver with Master: 37041af7ddbc4b7d8888b03c8a1a17ce (127.8.218.193:38653)
I20260928 10:45:32.551640 19422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:38727
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:45:32.585651 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36423
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:32.918354 19617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:32.918751 19617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:32.918908 19617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:32.930244 19617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:32.930380 19617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:32.930536 19617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:45:32.940770 19617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:32.942440 19617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:32.943905 19617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:32.957347 19622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:32.957404 19625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:32.957376 19623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:32.958137 19617 server_base.cc:1034] running on GCE node
I20260928 10:45:32.958739 19617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:32.959786 19617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:32.960994 19617 hybrid_clock.cc:648] HybridClock initialized: now 1790592332960927 us; error 61 us; skew 500 ppm
I20260928 10:45:32.961301 19617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:32 dist-test-slave-mn38 krb5kdc[19377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592332, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:32.964594 19617 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:45:32.966236 19617 webserver.cc:460] Webserver started at http://127.8.218.194:43181/ using document root <none> and password file <none>
I20260928 10:45:32.966982 19617 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:32.967099 19617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:32.967409 19617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:32.970666 19617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "84b6778ea84a47afa76f9e8d82e3a704"
format_stamp: "Formatted at 2026-09-28 10:45:32 on dist-test-slave-mn38"
server_key: "e61cbfb1fb319ce6f886eb19d949ae6f"
server_key_iv: "e0fad5ad9c91edf05e954892fd89026f"
server_key_version: "encryptionkey@0"
I20260928 10:45:32.971486 19617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "84b6778ea84a47afa76f9e8d82e3a704"
format_stamp: "Formatted at 2026-09-28 10:45:32 on dist-test-slave-mn38"
server_key: "e61cbfb1fb319ce6f886eb19d949ae6f"
server_key_iv: "e0fad5ad9c91edf05e954892fd89026f"
server_key_version: "encryptionkey@0"
I20260928 10:45:32.977310 19617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 10:45:32.981086 19632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:32.982712 19617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 10:45:32.982894 19617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "84b6778ea84a47afa76f9e8d82e3a704"
format_stamp: "Formatted at 2026-09-28 10:45:32 on dist-test-slave-mn38"
server_key: "e61cbfb1fb319ce6f886eb19d949ae6f"
server_key_iv: "e0fad5ad9c91edf05e954892fd89026f"
server_key_version: "encryptionkey@0"
I20260928 10:45:32.983114 19617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:33.001291 19617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:33.006275 19617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:33.006584 19617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:33.007838 19617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:33.009411 19617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:33.009534 19617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 10:45:33.009648 19617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:33.009707 19617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:33.040724 19617 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:33283
I20260928 10:45:33.040763 19744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:33283 every 8 connection(s)
I20260928 10:45:33.042088 19617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:45:33.048221 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 19617
I20260928 10:45:33.048382 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:45:33.048842 9067 external_mini_cluster.cc:1428] Setting key cc36959bd11bb6ccd2acc133f3638445
Sep 28 10:45:33 dist-test-slave-mn38 krb5kdc[19377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592332, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:45:33.063225 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.044598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:38485 (local address 127.8.218.254:36423)
0928 10:45:33.045119 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.045125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.046078 (+ 953us) server_negotiation.cc:407] Connection header received
0928 10:45:33.047882 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.047889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.047972 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.048066 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:33.049750 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.050519 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.052094 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.052516 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.052644 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.056428 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:33.056434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:33.056437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:33.056473 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:33.059740 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:33.060721 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:33.060726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:33.060729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:33.060868 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:33.061385 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:33.061389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:33.061391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:33.061741 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:33.061864 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.062759 (+ 895us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.062941 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
I20260928 10:45:33.064648 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.045189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36423 (local address 127.8.218.194:38485)
0928 10:45:33.045923 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.045969 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.047599 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.048321 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.048329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.048901 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:33.049578 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.049599 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.050673 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.050677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.051949 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.051961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.052745 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.052753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.052989 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.053709 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:33.053734 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:33.056149 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:33.059922 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:33.059932 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:33.059952 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:33.060553 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:33.060970 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:33.060974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:33.060978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:33.061253 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:33.061868 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:33.061876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:33.062374 (+ 498us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.062675 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.063240 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":276,"threads_started":1}
I20260928 10:45:33.066224 19745 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36423
I20260928 10:45:33.066531 19745 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:33.067303 19745 heartbeater.cc:511] Master 127.8.218.254:36423 requested a full tablet report, sending...
I20260928 10:45:33.069017 19422 ts_manager.cc:194] Registered new tserver with Master: 84b6778ea84a47afa76f9e8d82e3a704 (127.8.218.194:33283)
I20260928 10:45:33.069890 19422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:38485
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:45:33.101168 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36423
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:45:33.427779 19751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:45:33.428076 19751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:45:33.428171 19751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:45:33.435751 19751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:45:33.435884 19751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:45:33.436002 19751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:45:33.446257 19751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:45:33.447552 19751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:45:33.448807 19751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:45:33.462338 19756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:45:33.462316 19757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:33.462749 19751 server_base.cc:1034] running on GCE node
W20260928 10:45:33.462930 19759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:45:33.463620 19751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:45:33.464762 19751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:45:33.465968 19751 hybrid_clock.cc:648] HybridClock initialized: now 1790592333465908 us; error 49 us; skew 500 ppm
I20260928 10:45:33.466322 19751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:45:33 dist-test-slave-mn38 krb5kdc[19377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592333, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:45:33.469699 19751 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:45:33.471271 19751 webserver.cc:460] Webserver started at http://127.8.218.195:45739/ using document root <none> and password file <none>
I20260928 10:45:33.471998 19751 fs_manager.cc:359] Metadata directory not provided
I20260928 10:45:33.472107 19751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:45:33.472424 19751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:45:33.475534 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "928f26ace16449a4bb4857aebef62604"
format_stamp: "Formatted at 2026-09-28 10:45:33 on dist-test-slave-mn38"
server_key: "a7ef0675a2f11819073dfc34c08eadc2"
server_key_iv: "91af0a28d348e844e5d3a4c621cae1c7"
server_key_version: "encryptionkey@0"
I20260928 10:45:33.476332 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "928f26ace16449a4bb4857aebef62604"
format_stamp: "Formatted at 2026-09-28 10:45:33 on dist-test-slave-mn38"
server_key: "a7ef0675a2f11819073dfc34c08eadc2"
server_key_iv: "91af0a28d348e844e5d3a4c621cae1c7"
server_key_version: "encryptionkey@0"
I20260928 10:45:33.481942 19751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 10:45:33.485802 19766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:33.487349 19751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:45:33.487529 19751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "928f26ace16449a4bb4857aebef62604"
format_stamp: "Formatted at 2026-09-28 10:45:33 on dist-test-slave-mn38"
server_key: "a7ef0675a2f11819073dfc34c08eadc2"
server_key_iv: "91af0a28d348e844e5d3a4c621cae1c7"
server_key_version: "encryptionkey@0"
I20260928 10:45:33.487732 19751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:45:33.512387 19751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:45:33.516867 19751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:45:33.517184 19751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:45:33.518253 19751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:45:33.519793 19751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:45:33.519868 19751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:33.519966 19751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:45:33.520015 19751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:45:33.550606 19751 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:44299
I20260928 10:45:33.550709 19878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:44299 every 8 connection(s)
I20260928 10:45:33.551883 19751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:45:33.553251 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 19751
I20260928 10:45:33.553414 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:45:33.553993 9067 external_mini_cluster.cc:1428] Setting key 8dc52c5f88db32332d17d61eeaa487e8
I20260928 10:45:33.555143 19611 heartbeater.cc:503] Master 127.8.218.254:36423 was elected leader, sending a full tablet report...
Sep 28 10:45:33 dist-test-slave-mn38 krb5kdc[19377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592333, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:45:33.575856 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.554747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:37149 (local address 127.8.218.254:36423)
0928 10:45:33.554935 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.554941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.556457 (+ 1516us) server_negotiation.cc:407] Connection header received
0928 10:45:33.558513 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.558516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.558599 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.558682 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:33.560734 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.561566 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.563989 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.564442 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.564600 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.569531 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:33.569538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:33.569543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:33.569589 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:33.572108 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:33.573270 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:33.573275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:33.573277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:33.573421 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:33.573976 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:33.573980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:33.573982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:33.574271 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:33.574402 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.575389 (+ 987us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.575591 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 10:45:33.577257 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.555293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36423 (local address 127.8.218.195:37149)
0928 10:45:33.556305 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.556352 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.558270 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.558995 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.559006 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.559744 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:45:33.560568 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.560594 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.561722 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.561728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.563806 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.563825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.564712 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.564721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.565044 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.565912 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:45:33.565947 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:45:33.569233 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:45:33.572274 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:33.572285 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:33.572306 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:33.573109 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:33.573563 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:45:33.573569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:45:33.573573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:45:33.573859 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:45:33.574432 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:45:33.574442 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:45:33.574966 (+ 524us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.575327 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.575977 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":482,"threads_started":1}
I20260928 10:45:33.578744 19879 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36423
I20260928 10:45:33.579101 19879 heartbeater.cc:464] Registering TS with master...
I20260928 10:45:33.580004 19879 heartbeater.cc:511] Master 127.8.218.254:36423 requested a full tablet report, sending...
I20260928 10:45:33.581753 19421 ts_manager.cc:194] Registered new tserver with Master: 928f26ace16449a4bb4857aebef62604 (127.8.218.195:44299)
I20260928 10:45:33.582316 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:45:33.582912 19421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:37149
I20260928 10:45:33.602799 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.587111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.8.218.254:36423)
0928 10:45:33.587293 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.587297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.588971 (+ 1674us) server_negotiation.cc:407] Connection header received
0928 10:45:33.589128 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.589134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.589228 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.589341 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:33.590801 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.592070 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.593419 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.594077 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.594221 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.596448 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:33.596453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:33.596456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:33.596494 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:33.599629 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:33.600358 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:33.600364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:33.600368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:33.600547 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:33.601044 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:33.601050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:33.601053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:33.601616 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:33.601879 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.602190 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.602433 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 10:45:33.607172 19421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:45:33.614171 19421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:45:33.637298 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.626240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.8.218.194:33283)
0928 10:45:33.627529 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.627538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.628727 (+ 1189us) server_negotiation.cc:407] Connection header received
0928 10:45:33.629172 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.629181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.629473 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.629662 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:33.631058 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.632592 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.635057 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.636130 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.636256 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.636462 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.636777 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.636950 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":1023,"threads_started":1}
I20260928 10:45:33.637631 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.627000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.8.218.195:44299)
0928 10:45:33.628002 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.628010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.629347 (+ 1337us) server_negotiation.cc:407] Connection header received
0928 10:45:33.629695 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.629703 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.629936 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.630089 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:33.631155 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.632294 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.635009 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.636270 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.636401 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.636614 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.637112 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.637282 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":669,"threads_started":1}
I20260928 10:45:33.638038 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.628204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:38653 (local address 127.0.0.1:48826)
0928 10:45:33.629630 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.629646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.629734 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.631004 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.631008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.631031 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:33.631358 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.631368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.633188 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.633193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.635303 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.635315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.637085 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.637091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.637254 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.637285 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.637375 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.637616 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"mutex_wait_us":1,"thread_start_us":1186,"threads_started":1}
I20260928 10:45:33.638294 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.628820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.8.218.193:38653)
0928 10:45:33.630272 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.630280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.630305 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:45:33.630389 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.630395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.630599 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.630772 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:33.631553 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.633042 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.635458 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.636869 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.637019 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.637241 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.637422 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.637845 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291,"thread_start_us":1117,"threads_started":1}
I20260928 10:45:33.638562 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.626023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33283 (local address 127.0.0.1:55088)
0928 10:45:33.628579 (+ 2556us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.628620 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.628881 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.630321 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.630328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.630355 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:33.630859 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.630874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.632775 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.632781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.634919 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.634934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.636316 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.636323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.636506 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.636550 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.637677 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.637946 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"mutex_wait_us":69,"thread_start_us":2120,"threads_started":1}
I20260928 10:45:33.638777 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.626396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:44299 (local address 127.0.0.1:59740)
0928 10:45:33.629221 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.629243 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.629484 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.630399 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.630403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.630437 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:33.630936 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.630950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.632431 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.632435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.634788 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.634804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.636759 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.636766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.636919 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.636951 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.637826 (+ 875us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.637980 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2700,"mutex_wait_us":42,"thread_start_us":2579,"threads_started":1}
I20260928 10:45:33.641937 19546 tablet_service.cc:1467] Processing CreateTablet for tablet 014984cee3bf4f739122af61c2b23f93 (DEFAULT_TABLE table=test-table [id=3d549e0330124583b36b0ca6ff29d50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:33.642091 19680 tablet_service.cc:1467] Processing CreateTablet for tablet 014984cee3bf4f739122af61c2b23f93 (DEFAULT_TABLE table=test-table [id=3d549e0330124583b36b0ca6ff29d50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:33.643189 19546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014984cee3bf4f739122af61c2b23f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:33.642864 19814 tablet_service.cc:1467] Processing CreateTablet for tablet 014984cee3bf4f739122af61c2b23f93 (DEFAULT_TABLE table=test-table [id=3d549e0330124583b36b0ca6ff29d50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:45:33.643401 19680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014984cee3bf4f739122af61c2b23f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:33.644066 19814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014984cee3bf4f739122af61c2b23f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:45:33.655071 19897 tablet_bootstrap.cc:492] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Bootstrap starting.
I20260928 10:45:33.659819 19897 tablet_bootstrap.cc:654] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:33.660815 19899 tablet_bootstrap.cc:492] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Bootstrap starting.
I20260928 10:45:33.660964 19897 log.cc:824] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:33.664292 19897 tablet_bootstrap.cc:492] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: No bootstrap required, opened a new log
I20260928 10:45:33.664619 19897 ts_tablet_manager.cc:1397] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 10:45:33.664896 19899 tablet_bootstrap.cc:654] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:33.666261 19899 log.cc:824] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:33.666280 19898 tablet_bootstrap.cc:492] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Bootstrap starting.
I20260928 10:45:33.670456 19898 tablet_bootstrap.cc:654] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Neither blocks nor log segments found. Creating new log.
I20260928 10:45:33.671924 19898 log.cc:824] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Log is configured to *not* fsync() on all Append() calls
I20260928 10:45:33.675557 19899 tablet_bootstrap.cc:492] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: No bootstrap required, opened a new log
I20260928 10:45:33.675668 19898 tablet_bootstrap.cc:492] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: No bootstrap required, opened a new log
I20260928 10:45:33.675864 19899 ts_tablet_manager.cc:1397] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260928 10:45:33.675956 19898 ts_tablet_manager.cc:1397] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 10:45:33.675890 19897 raft_consensus.cc:348] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.676409 19897 raft_consensus.cc:374] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:33.676532 19897 raft_consensus.cc:729] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37041af7ddbc4b7d8888b03c8a1a17ce, State: Initialized, Role: FOLLOWER
I20260928 10:45:33.677345 19897 consensus_queue.cc:260] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.683158 19897 ts_tablet_manager.cc:1428] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260928 10:45:33.684763 19899 raft_consensus.cc:348] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.685286 19899 raft_consensus.cc:374] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:33.685391 19899 raft_consensus.cc:729] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928f26ace16449a4bb4857aebef62604, State: Initialized, Role: FOLLOWER
I20260928 10:45:33.686137 19898 raft_consensus.cc:348] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.686223 19899 consensus_queue.cc:260] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.686501 19898 raft_consensus.cc:374] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:45:33.686591 19898 raft_consensus.cc:729] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b6778ea84a47afa76f9e8d82e3a704, State: Initialized, Role: FOLLOWER
I20260928 10:45:33.687151 19898 consensus_queue.cc:260] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.688270 19745 heartbeater.cc:503] Master 127.8.218.254:36423 was elected leader, sending a full tablet report...
I20260928 10:45:33.689088 19898 ts_tablet_manager.cc:1428] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 10:45:33.690460 19879 heartbeater.cc:503] Master 127.8.218.254:36423 was elected leader, sending a full tablet report...
I20260928 10:45:33.691221 19899 ts_tablet_manager.cc:1428] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Time spent starting tablet: real 0.015s user 0.004s sys 0.012s
W20260928 10:45:33.773021 19612 tablet.cc:2369] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:45:33.791849 19904 raft_consensus.cc:482] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:45:33.792166 19904 raft_consensus.cc:504] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.794051 19904 leader_election.cc:288] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84b6778ea84a47afa76f9e8d82e3a704 (127.8.218.194:33283), 37041af7ddbc4b7d8888b03c8a1a17ce (127.8.218.193:38653)
W20260928 10:45:33.794119 19746 tablet.cc:2369] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:45:33.800678 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.794543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:35605 (local address 127.8.218.194:33283)
0928 10:45:33.794736 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.794741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.794762 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:45:33.794838 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.794842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.794920 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.795021 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:33.795465 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.796485 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.798530 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.799731 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.799862 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.799918 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.800292 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.800404 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:45:33.800670 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.794247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33283 (local address 127.8.218.195:35605)
0928 10:45:33.794551 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.794573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.794700 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.795055 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.795058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.795086 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:33.795270 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.795278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.796611 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.796616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.798310 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.798323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.799889 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.799895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.800081 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.800116 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.800238 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.800379 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 10:45:33.801638 19903 raft_consensus.cc:482] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:45:33.802009 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.794247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:38653 (local address 127.8.218.195:46659)
0928 10:45:33.795479 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.795505 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.795642 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.795917 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.795922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.795947 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:33.796310 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.796321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.797741 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.797745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.799604 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.799620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.801196 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.801202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.801364 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.801391 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.801515 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.801650 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":546,"thread_start_us":563,"threads_started":1}
I20260928 10:45:33.801834 19700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014984cee3bf4f739122af61c2b23f93" candidate_uuid: "928f26ace16449a4bb4857aebef62604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b6778ea84a47afa76f9e8d82e3a704" is_pre_election: true
I20260928 10:45:33.801956 19903 raft_consensus.cc:504] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.802042 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.794681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:46659 (local address 127.8.218.193:38653)
0928 10:45:33.794879 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.794884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.795614 (+ 730us) server_negotiation.cc:407] Connection header received
0928 10:45:33.795751 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.795755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.795813 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.795882 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:33.796465 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.797557 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.799794 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.801004 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.801121 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.801174 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.801555 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.801696 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:45:33.802399 19700 raft_consensus.cc:2454] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928f26ace16449a4bb4857aebef62604 in term 0.
I20260928 10:45:33.803153 19769 leader_election.cc:302] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84b6778ea84a47afa76f9e8d82e3a704, 928f26ace16449a4bb4857aebef62604; no voters:
W20260928 10:45:33.803462 19880 tablet.cc:2369] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:45:33.803301 19566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014984cee3bf4f739122af61c2b23f93" candidate_uuid: "928f26ace16449a4bb4857aebef62604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" is_pre_election: true
I20260928 10:45:33.803766 19566 raft_consensus.cc:2454] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928f26ace16449a4bb4857aebef62604 in term 0.
I20260928 10:45:33.803821 19904 raft_consensus.cc:2781] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:45:33.804060 19904 raft_consensus.cc:482] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:45:33.804162 19903 leader_election.cc:288] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84b6778ea84a47afa76f9e8d82e3a704 (127.8.218.194:33283), 928f26ace16449a4bb4857aebef62604 (127.8.218.195:44299)
I20260928 10:45:33.804230 19904 raft_consensus.cc:3037] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:33.807447 19904 raft_consensus.cc:504] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.808609 19700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014984cee3bf4f739122af61c2b23f93" candidate_uuid: "928f26ace16449a4bb4857aebef62604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b6778ea84a47afa76f9e8d82e3a704"
I20260928 10:45:33.808728 19566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014984cee3bf4f739122af61c2b23f93" candidate_uuid: "928f26ace16449a4bb4857aebef62604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce"
I20260928 10:45:33.808857 19700 raft_consensus.cc:3037] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:33.808947 19566 raft_consensus.cc:3037] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:45:33.812286 19566 raft_consensus.cc:2454] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928f26ace16449a4bb4857aebef62604 in term 1.
I20260928 10:45:33.812291 19700 raft_consensus.cc:2454] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928f26ace16449a4bb4857aebef62604 in term 1.
I20260928 10:45:33.812633 19904 leader_election.cc:288] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [CANDIDATE]: Term 1 election: Requested vote from peers 84b6778ea84a47afa76f9e8d82e3a704 (127.8.218.194:33283), 37041af7ddbc4b7d8888b03c8a1a17ce (127.8.218.193:38653)
I20260928 10:45:33.812986 19769 leader_election.cc:302] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84b6778ea84a47afa76f9e8d82e3a704, 928f26ace16449a4bb4857aebef62604; no voters:
I20260928 10:45:33.813262 19904 raft_consensus.cc:2781] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:45:33.813833 19904 raft_consensus.cc:686] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 1 LEADER]: Becoming Leader. State: Replica: 928f26ace16449a4bb4857aebef62604, State: Running, Role: LEADER
I20260928 10:45:33.814409 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.804886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:35137 (local address 127.8.218.195:44299)
0928 10:45:33.805186 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.805191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.805627 (+ 436us) server_negotiation.cc:407] Connection header received
0928 10:45:33.805943 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.805949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.806047 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.806118 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:33.806914 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.808117 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.811572 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.812876 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.813030 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.813109 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.814000 (+ 891us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.814149 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260928 10:45:33.814424 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.805584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:57825 (local address 127.8.218.194:33283)
0928 10:45:33.805765 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.805771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.805925 (+ 154us) server_negotiation.cc:407] Connection header received
0928 10:45:33.806183 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.806188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.806262 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.806346 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:45:33.807009 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.808134 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.811563 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.812905 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.813040 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.813109 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.814012 (+ 903us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.814149 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:45:33.814710 19904 consensus_queue.cc:237] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } }
I20260928 10:45:33.815169 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.804963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33283 (local address 127.8.218.193:57825)
0928 10:45:33.805832 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.805849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.806024 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.806382 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.806386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.806417 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:33.806836 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.806848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.808288 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.808293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.811327 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.811350 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.813376 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.813389 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.813756 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.813837 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.814584 (+ 747us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.814742 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":306,"thread_start_us":719,"threads_started":1}
I20260928 10:45:33.815201 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.804784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:44299 (local address 127.8.218.193:35137)
0928 10:45:33.805519 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 10:45:33.805539 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:45:33.805769 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:45:33.806313 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:45:33.806318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:45:33.806345 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:45:33.806770 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.806783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.808283 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.808289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.811327 (+ 3038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:45:33.811350 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.813400 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.813405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:45:33.813752 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.813837 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 10:45:33.814703 (+ 866us) client_negotiation.cc:240] Negotiation successful
0928 10:45:33.814822 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":255,"thread_start_us":411,"threads_started":1}
I20260928 10:45:33.816164 19834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014984cee3bf4f739122af61c2b23f93" candidate_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "928f26ace16449a4bb4857aebef62604" is_pre_election: true
I20260928 10:45:33.816771 19700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014984cee3bf4f739122af61c2b23f93" candidate_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b6778ea84a47afa76f9e8d82e3a704" is_pre_election: true
I20260928 10:45:33.817121 19700 raft_consensus.cc:2379] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37041af7ddbc4b7d8888b03c8a1a17ce in current term 1: Already voted for candidate 928f26ace16449a4bb4857aebef62604 in this term.
I20260928 10:45:33.817970 19501 leader_election.cc:302] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37041af7ddbc4b7d8888b03c8a1a17ce; no voters: 84b6778ea84a47afa76f9e8d82e3a704, 928f26ace16449a4bb4857aebef62604
I20260928 10:45:33.818356 19903 raft_consensus.cc:2726] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 10:45:33.820607 19421 catalog_manager.cc:5495] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 928f26ace16449a4bb4857aebef62604 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "928f26ace16449a4bb4857aebef62604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "928f26ace16449a4bb4857aebef62604" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 44299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 } health_report { overall_health: UNKNOWN } } }
I20260928 10:45:33.877650 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.872316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.8.218.195:44299)
0928 10:45:33.872494 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.872500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.872559 (+ 59us) server_negotiation.cc:407] Connection header received
0928 10:45:33.872792 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.872796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.872875 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.873002 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:45:33.873384 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.874157 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.875453 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.875863 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.875981 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.876478 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:45:33.877005 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:45:33.877146 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.877258 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.877341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:45:33.885910 19700 raft_consensus.cc:1261] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 1 FOLLOWER]: Refusing update from remote peer 928f26ace16449a4bb4857aebef62604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:33.885937 19566 raft_consensus.cc:1261] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 1 FOLLOWER]: Refusing update from remote peer 928f26ace16449a4bb4857aebef62604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:45:33.886823 19909 consensus_queue.cc:1035] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b6778ea84a47afa76f9e8d82e3a704" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:33.887187 19904 consensus_queue.cc:1035] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37041af7ddbc4b7d8888b03c8a1a17ce" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:45:33.900673 19914 mvcc.cc:205] Tried to move back new op lower bound from 7334266199586320384 to 7334266199309647872. Current Snapshot: MvccSnapshot[applied={T|T < 7334266199586320384}]
I20260928 10:45:33.913183 19421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:33.913491 19421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:45:33.917629 19421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc7bfa8cefce463ead143770f5156127: Sending DeleteTablet for 3 replicas of tablet 014984cee3bf4f739122af61c2b23f93
I20260928 10:45:33.918814 19814 tablet_service.cc:1514] Processing DeleteTablet for tablet 014984cee3bf4f739122af61c2b23f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:33 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:59740
I20260928 10:45:33.918881 19680 tablet_service.cc:1514] Processing DeleteTablet for tablet 014984cee3bf4f739122af61c2b23f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:33 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:55088
I20260928 10:45:33.919971 19546 tablet_service.cc:1514] Processing DeleteTablet for tablet 014984cee3bf4f739122af61c2b23f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:45:33 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:48826
I20260928 10:45:33.921159 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 19483
I20260928 10:45:33.926185 19922 tablet_replica.cc:330] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: stopping tablet replica
I20260928 10:45:33.927134 19922 raft_consensus.cc:2229] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:45:33.928398 19922 raft_consensus.cc:2258] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:33.929916 19924 tablet_replica.cc:330] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: stopping tablet replica
I20260928 10:45:33.930287 19924 raft_consensus.cc:2229] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:33.930737 19924 raft_consensus.cc:2258] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:33.935283 19922 ts_tablet_manager.cc:1905] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:33.935690 19924 ts_tablet_manager.cc:1905] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:45:33.942826 19922 ts_tablet_manager.cc:1918] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:33.942802 19924 ts_tablet_manager.cc:1918] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:33.942994 19924 log.cc:1196] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/014984cee3bf4f739122af61c2b23f93
I20260928 10:45:33.943188 19922 log.cc:1196] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/014984cee3bf4f739122af61c2b23f93
I20260928 10:45:33.943518 19924 ts_tablet_manager.cc:1939] T 014984cee3bf4f739122af61c2b23f93 P 37041af7ddbc4b7d8888b03c8a1a17ce: Deleting consensus metadata
I20260928 10:45:33.943881 19922 ts_tablet_manager.cc:1939] T 014984cee3bf4f739122af61c2b23f93 P 928f26ace16449a4bb4857aebef62604: Deleting consensus metadata
I20260928 10:45:33.945580 19406 catalog_manager.cc:4841] TS 37041af7ddbc4b7d8888b03c8a1a17ce (127.8.218.193:38653): tablet 014984cee3bf4f739122af61c2b23f93 (table test-table [id=3d549e0330124583b36b0ca6ff29d50f]) successfully deleted
I20260928 10:45:33.945561 19407 catalog_manager.cc:4841] TS 928f26ace16449a4bb4857aebef62604 (127.8.218.195:44299): tablet 014984cee3bf4f739122af61c2b23f93 (table test-table [id=3d549e0330124583b36b0ca6ff29d50f]) successfully deleted
I20260928 10:45:33.947489 19923 tablet_replica.cc:330] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: stopping tablet replica
I20260928 10:45:33.947958 19923 raft_consensus.cc:2229] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:45:33.948513 19923 raft_consensus.cc:2258] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:45:33.951260 19923 ts_tablet_manager.cc:1905] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 10:45:33 dist-test-slave-mn38 krb5kdc[19377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:45:33.959183 19923 ts_tablet_manager.cc:1918] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:45:33.959396 19923 log.cc:1196] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/014984cee3bf4f739122af61c2b23f93
I20260928 10:45:33.959991 19923 ts_tablet_manager.cc:1939] T 014984cee3bf4f739122af61c2b23f93 P 84b6778ea84a47afa76f9e8d82e3a704: Deleting consensus metadata
I20260928 10:45:33.961534 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:33.922497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.8.218.193:38653)
0928 10:45:33.922671 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:33.922678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:33.945914 (+ 23236us) server_negotiation.cc:407] Connection header received
0928 10:45:33.946142 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:33.946147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:33.946235 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:33.946334 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:33.948028 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.949180 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.950600 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:33.951006 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:33.951121 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:33.955258 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:33.955273 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:33.955289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:33.955329 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:33.958254 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:33.959082 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:33.959091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:33.959093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:33.959229 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:33.959825 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:33.959829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:33.959831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:33.960269 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:33.960566 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:33.960910 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 10:45:33.961152 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:45:33.961966 19408 catalog_manager.cc:4841] TS 84b6778ea84a47afa76f9e8d82e3a704 (127.8.218.194:33283): tablet 014984cee3bf4f739122af61c2b23f93 (table test-table [id=3d549e0330124583b36b0ca6ff29d50f]) successfully deleted
I20260928 10:45:33.962162 19606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:34.059778 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 19483
I20260928 10:45:34.078119 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 19617
Sep 28 10:45:34 dist-test-slave-mn38 krb5kdc[19377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:45:34.095634 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:34.079001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.8.218.194:33283)
0928 10:45:34.079174 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:34.079179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:34.079197 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:45:34.079334 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:34.079338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:34.079415 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:34.079499 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:34.081058 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:34.081893 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:34.083715 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:34.084159 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:34.084257 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:34.089431 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:34.089483 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:34.089503 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:34.089560 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:34.092671 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:34.093488 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:34.093497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:34.093500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:34.093652 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:34.094050 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:34.094055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:34.094059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:34.094444 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:34.094743 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:34.095074 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 10:45:34.095298 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 10:45:34.096395 19740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:34.186970 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 19617
I20260928 10:45:34.202185 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 19751
Sep 28 10:45:34 dist-test-slave-mn38 krb5kdc[19377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:45:34.222661 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:45:34.203059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.8.218.195:44299)
0928 10:45:34.203312 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 10:45:34.203319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:45:34.203379 (+ 60us) server_negotiation.cc:407] Connection header received
0928 10:45:34.203454 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:45:34.203460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:45:34.203542 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:45:34.203667 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:45:34.205049 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:34.206281 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:34.207987 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:45:34.208698 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:45:34.208875 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:45:34.214404 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:45:34.214427 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:45:34.214449 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:45:34.214515 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:45:34.219158 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:34.220090 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:34.220106 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:34.220110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:34.220291 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:45:34.220726 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:45:34.220731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:45:34.220734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:45:34.221271 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:45:34.221632 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 10:45:34.222050 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 10:45:34.222300 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 10:45:34.223433 19874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:34.345013 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 19751
I20260928 10:45:34.360785 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 19390
I20260928 10:45:34.361950 19452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:45:34.470774 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 19390
2026-09-28T10:45:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4303 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:45:34 dist-test-slave-mn38 krb5kdc[19932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:45:34 dist-test-slave-mn38 krb5kdc[19932](info): set up 2 sockets
Sep 28 10:45:34 dist-test-slave-mn38 krb5kdc[19932](info): commencing operation
krb5kdc: starting...
W20260928 10:45:35.491242 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:45:35 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:45:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.218.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.218.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/rangeradmin_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/rangeradmin_127.8.218.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.218.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.218.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/rangerlookup_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/rangerlookup_127.8.218.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.218.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.218.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/HTTP_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/HTTP_127.8.218.212@KRBTEST.COM.keytab.
I20260928 10:45:35.825685 9067 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3zclSv/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/postgres -l logfile start
2026-09-28 10:45:38.490 UTC [19962] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 10:45:38.490 UTC [19962] LOG: listening on IPv4 address "127.8.218.212", port 33249
2026-09-28 10:45:38.494 UTC [19962] LOG: listening on Unix socket "/tmp/.s.PGSQL.33249"
2026-09-28 10:45:38.516 UTC [19965] LOG: database system was shut down at 2026-09-28 10:45:36 UTC
2026-09-28 10:45:38.520 UTC [19962] LOG: database system is ready to accept connections
I20260928 10:45:39.446822 9067 mini_postgres.cc:96] Postgres bound to 33249
2026-09-28 10:45:39.464 UTC [19973] FATAL: database "slave" does not exist
127.8.218.212:33249 - accepting connections
I20260928 10:45:39.464906 9067 mini_ranger.cc:161] Starting Ranger...
I20260928 10:45:39.491710 9067 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 10:45:40.121549 9067 mini_ranger.cc:87] Created ranger Postgres database
I20260928 10:45:40.121711 9067 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-admin
2026-09-28 10:45:40,409 [I] DB FLAVOR :POSTGRES
2026-09-28 10:45:40,410 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:45:40,410 [I] Checking connection..
2026-09-28 10:45:40,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 10:45:40,684 [I] Checking connection passed.
2026-09-28 10:45:40,684 [I] --------- Verifying version history table ---------
2026-09-28 10:45:40,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:45:40,931 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:45:40,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:45:41,203 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:45:41,204 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 10:45:41,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 10:45:41.475 UTC [20037] WARNING: there is no transaction in progress
2026-09-28 10:45:41,481 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 10:45:41,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:45:41,734 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 10:45:41,735 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:45:41,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 10:45:41,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 10:45:42,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mn38.c.gcp-upstream.internal','N') ;"
2026-09-28 10:45:42,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:45:42,751 [I] Table x_portal_user does not exist in database ranger
2026-09-28 10:45:42,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:45:42,995 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 10:45:42,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:45:43,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:45:43,511 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 10:45:43,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 10:45:44.578 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.599 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.616 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.634 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.651 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.670 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.729 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.736 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.751 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.763 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.775 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.791 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.802 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.809 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.818 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:44.827 UTC [20145] WARNING: there is no transaction in progress
2026-09-28 10:45:45,578 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 10:45:45,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 10:45:45,808 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 10:45:45,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:45:46,058 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 10:45:46,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:45:46,326 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 10:45:46,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:45:46,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:45:46,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mn38.c.gcp-upstream.internal';"
2026-09-28 10:45:47,093 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 10:45:47,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 10:45:47,339 [I] DB_PATCHES have already been applied
I20260928 10:45:47.345641 9067 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-admin:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3zclSv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 10:45:47.345750 9067 mini_ranger.cc:193] Using host: 127.8.218.212
I20260928 10:45:47.359781 9067 mini_ranger.cc:227] Ranger admin URL: http://127.8.218.212:56291
Sep 28, 2026 10:45:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:45:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:45:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:45:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:45:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:45:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:45:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 10:45:48.380668 9067 mini_ranger.cc:160] Time spent starting Ranger: real 8.916s user 0.012s sys 0.028s
Sep 28, 2026 10:46:08 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:46:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 10:46:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:46:08 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:46:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:46:09 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 10:46:10.533339 9067 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.218.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.218.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/rangerkms_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/rangerkms_127.8.218.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.218.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.218.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.218.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/HTTP_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.218.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/HTTP_127.8.218.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 10:46:10 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 10:46:10.719776 9067 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 10:46:10.744220 9067 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 10:46:11.718086 9067 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 10:46:11.718197 9067 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms
I20260928 10:46:11.718336 9067 mini_ranger_kms.cc:227] Using postgres at 127.8.218.212:33249
2026-09-28 10:46:12,129 [I] DB FLAVOR :POSTGRES
2026-09-28 10:46:12,129 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:46:12,129 [I] Checking connection
2026-09-28 10:46:12,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 10:46:12,373 [I] connection success
2026-09-28 10:46:12,373 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 10:46:12,373 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 10:46:12,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 10:46:12,636 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 10:46:12,636 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:46:12,636 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 10:46:12,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:33249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 10:46:12,907 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 10:46:13.443326 9067 mini_ranger_kms.cc:310] Created kms service
I20260928 10:46:13.669879 9067 mini_ranger_kms.cc:326] Created kudu user
I20260928 10:46:13.721120 9067 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 10:46:13 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592370, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:46:14.152527 9067 mini_ranger_kms.cc:384] Added ranger policy
I20260928 10:46:14.152874 9067 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms:/tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 10:46:14.152943 9067 mini_ranger_kms.cc:240] Using host: 127.8.218.212
I20260928 10:46:14.178646 9067 mini_ranger_kms.cc:276] Ranger KMS PID: 20326
I20260928 10:46:14.178882 9067 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.218.212:34219
Sep 28, 2026 10:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:46:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 10:46:16.331377 9067 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.612s user 0.118s sys 0.323s
I20260928 10:46:16.331617 9067 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 10:46:16.331816 9067 mini_ranger_kms.cc:401] 127.8.218.212:34219/kms/v1/keys
Sep 28 10:46:16 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.218.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 10:46:16 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.218.212@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
Sep 28 10:46:17 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592376, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.218.212@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
Sep 28 10:46:19 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:46:19.501590 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:44205
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:44205
--ranger_config_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:34219/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:46:19.830310 20376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:46:19.830629 20376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:46:19.830716 20376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:46:19.838121 20376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:46:19.838246 20376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:46:19.838289 20376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:46:19.838346 20376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:46:19.838392 20376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:46:19.848594 20376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.218.212:34219/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:44205
--ranger_config_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:46:19.849884 20376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:46:19.851115 20376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:46:19.863247 20382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:46:19.863412 20381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:46:19.863404 20384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:46:19.864051 20376 server_base.cc:1034] running on GCE node
I20260928 10:46:19.864809 20376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:46:19.866361 20376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:46:19.867563 20376 hybrid_clock.cc:648] HybridClock initialized: now 1790592379867509 us; error 49 us; skew 500 ppm
I20260928 10:46:19.867899 20376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:46:19 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592379, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:46:19.874090 20376 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:46:19.875698 20376 webserver.cc:460] Webserver started at http://127.8.218.254:43287/ using document root <none> and password file <none>
I20260928 10:46:19.876339 20376 fs_manager.cc:359] Metadata directory not provided
I20260928 10:46:19.876434 20376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:46:19.876689 20376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:46:19 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592379, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:46:19.934978 20376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "e1e88c7d6ede40549403a305763aba18"
format_stamp: "Formatted at 2026-09-28 10:46:19 on dist-test-slave-mn38"
server_key: "05ef6454c4a065155d99f082e56571aa"
server_key_iv: "3307062ed2bf2fdfce514db9fc734e83"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:19.935869 20376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "e1e88c7d6ede40549403a305763aba18"
format_stamp: "Formatted at 2026-09-28 10:46:19 on dist-test-slave-mn38"
server_key: "05ef6454c4a065155d99f082e56571aa"
server_key_iv: "3307062ed2bf2fdfce514db9fc734e83"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:19.941216 20376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 10:46:19.962559 20391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:46:19.964187 20376 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:46:19.964365 20376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "e1e88c7d6ede40549403a305763aba18"
format_stamp: "Formatted at 2026-09-28 10:46:19 on dist-test-slave-mn38"
server_key: "05ef6454c4a065155d99f082e56571aa"
server_key_iv: "3307062ed2bf2fdfce514db9fc734e83"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:19.964574 20376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:46:19.987910 20376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:46:19.992010 20376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:46:19.992303 20376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:46:20.008883 20376 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:44205
I20260928 10:46:20.008886 20442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:44205 every 8 connection(s)
I20260928 10:46:20.010324 20376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:46:20.014269 20443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:46:20.017920 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 20376
I20260928 10:46:20.018112 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:46:20.018872 9067 external_mini_cluster.cc:1428] Setting key 2fc54e7eee8a4f3f77b3daa8cf4f5b80
I20260928 10:46:20.027580 20443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18: Bootstrap starting.
I20260928 10:46:20.031366 20443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18: Neither blocks nor log segments found. Creating new log.
I20260928 10:46:20.032680 20443 log.cc:824] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:46:20 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:46:20.035843 20443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18: No bootstrap required, opened a new log
I20260928 10:46:20.041658 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:20.020953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.8.218.254:44205)
0928 10:46:20.021748 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:20.021761 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:20.021803 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:46:20.023379 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:20.023403 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:20.023925 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:20.024356 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:46:20.026028 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:20.027364 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:20.029018 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:20.029780 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:20.029916 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:20.034422 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:46:20.034430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:46:20.034449 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:46:20.034491 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:46:20.037616 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:20.038312 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:20.038323 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:20.038330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:20.038497 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:20.038871 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:20.038875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:20.038877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:20.039405 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:46:20.039680 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:20.040161 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 10:46:20.040534 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":355,"threads_started":1}
I20260928 10:46:20.044071 20443 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e88c7d6ede40549403a305763aba18" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44205 } }
I20260928 10:46:20.044453 20443 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:46:20.044553 20443 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e88c7d6ede40549403a305763aba18, State: Initialized, Role: FOLLOWER
I20260928 10:46:20.045053 20443 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e88c7d6ede40549403a305763aba18" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44205 } }
I20260928 10:46:20.045252 20443 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:46:20.045353 20443 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:46:20.045496 20443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:46:20.047552 20443 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e88c7d6ede40549403a305763aba18" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44205 } }
I20260928 10:46:20.048017 20443 leader_election.cc:302] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e88c7d6ede40549403a305763aba18; no voters:
I20260928 10:46:20.048597 20443 leader_election.cc:288] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:46:20.048754 20448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:46:20.049679 20448 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [term 1 LEADER]: Becoming Leader. State: Replica: e1e88c7d6ede40549403a305763aba18, State: Running, Role: LEADER
I20260928 10:46:20.050218 20448 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e88c7d6ede40549403a305763aba18" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44205 } }
I20260928 10:46:20.050568 20443 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:46:20.053129 20449 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1e88c7d6ede40549403a305763aba18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e88c7d6ede40549403a305763aba18" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44205 } } }
I20260928 10:46:20.053550 20449 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [sys.catalog]: This master's current role is: LEADER
I20260928 10:46:20.053911 20450 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e88c7d6ede40549403a305763aba18. Latest consensus state: current_term: 1 leader_uuid: "e1e88c7d6ede40549403a305763aba18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e88c7d6ede40549403a305763aba18" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44205 } } }
I20260928 10:46:20.054234 20450 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18 [sys.catalog]: This master's current role is: LEADER
I20260928 10:46:20.056891 20457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:46:20.062129 20457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:46:20.062942 20443 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 10:46:20.079095 20457 catalog_manager.cc:1347] Generated new cluster ID: b6d6cc5bb7c8485193e3f7c4341d8e27
I20260928 10:46:20.079238 20457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:46:20.113881 20457 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:46:20.115267 20457 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:46:20.125339 20457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18: Generated new TSK 0
I20260928 10:46:20.126313 20457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:46:20.902925 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:34219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44205
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:46:21 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592381, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 10:46:21.326699 20488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:46:21.327098 20488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:46:21.327313 20488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:46:21.339236 20488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:46:21.339391 20488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:46:21.339627 20488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:46:21.355784 20488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.218.212:34219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:46:21.357744 20488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:46:21.359529 20488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:46:21.382484 20498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:46:21.383464 20499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:46:21.387809 20501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:46:21.388236 20488 server_base.cc:1034] running on GCE node
I20260928 10:46:21.389001 20488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:46:21.390070 20488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:46:21.391304 20488 hybrid_clock.cc:648] HybridClock initialized: now 1790592381391228 us; error 65 us; skew 500 ppm
I20260928 10:46:21.391736 20488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:46:21 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592381, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:46:21.398435 20488 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:46:21.400628 20488 webserver.cc:460] Webserver started at http://127.8.218.193:45483/ using document root <none> and password file <none>
I20260928 10:46:21.401770 20488 fs_manager.cc:359] Metadata directory not provided
I20260928 10:46:21.401973 20488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:46:21.402407 20488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:46:21 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592381, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:46:21.423377 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "14741664038a4e48aedccf863b39a7d9"
format_stamp: "Formatted at 2026-09-28 10:46:21 on dist-test-slave-mn38"
server_key: "fa1760964ae071f8981c461bd0ae2c9c"
server_key_iv: "71d75cec3b2d398a50cc98aa12822327"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:21.424672 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "14741664038a4e48aedccf863b39a7d9"
format_stamp: "Formatted at 2026-09-28 10:46:21 on dist-test-slave-mn38"
server_key: "fa1760964ae071f8981c461bd0ae2c9c"
server_key_iv: "71d75cec3b2d398a50cc98aa12822327"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:21.459949 20488 fs_manager.cc:691] Time spent creating directory manager: real 0.035s user 0.010s sys 0.023s
I20260928 10:46:21.480660 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:46:21.483887 20488 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260928 10:46:21.484174 20488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "14741664038a4e48aedccf863b39a7d9"
format_stamp: "Formatted at 2026-09-28 10:46:21 on dist-test-slave-mn38"
server_key: "fa1760964ae071f8981c461bd0ae2c9c"
server_key_iv: "71d75cec3b2d398a50cc98aa12822327"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:21.484526 20488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:46:21.559820 20488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:46:21.565366 20488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:46:21.565802 20488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:46:21.581909 20488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:46:21.599300 20488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:46:21.599520 20488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:46:21.599750 20488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:46:21.599880 20488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:46:21 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592381, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:46:21.901872 20488 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:33923
I20260928 10:46:21.901881 20620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:33923 every 8 connection(s)
I20260928 10:46:21.903688 20488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:46:21.905843 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 20488
I20260928 10:46:21.906046 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:46:21.906847 9067 external_mini_cluster.cc:1428] Setting key d03d4abc60ca5bd2b2366c31fa8406b6
Sep 28 10:46:21 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592381, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:46:21.981501 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:21.925185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44205 (local address 127.8.218.193:56051)
0928 10:46:21.945968 (+ 20783us) negotiation.cc:107] Waiting for socket to connect
0928 10:46:21.946035 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 10:46:21.948904 (+ 2869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:46:21.962003 (+ 13099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:46:21.962021 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:46:21.962853 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:46:21.963902 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:21.963933 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:21.966190 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:21.966195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:21.968094 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:21.968118 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:21.969327 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:21.969335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:21.969656 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:21.970774 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:46:21.970804 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:46:21.973303 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:46:21.976904 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:46:21.976913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:46:21.976932 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:46:21.977502 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:46:21.977938 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:46:21.977942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:46:21.977945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:46:21.978181 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:46:21.978809 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:46:21.978817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:46:21.979285 (+ 468us) client_negotiation.cc:769] Sending connection context
0928 10:46:21.979577 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 10:46:21.980108 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20364,"thread_start_us":27911,"threads_started":1}
I20260928 10:46:21.982540 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:21.925951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:56051 (local address 127.8.218.254:44205)
0928 10:46:21.945122 (+ 19171us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:21.945130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:21.949498 (+ 4368us) server_negotiation.cc:407] Connection header received
0928 10:46:21.949590 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:21.949596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:21.949718 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:21.949817 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:46:21.964823 (+ 15006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:21.965979 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:21.968286 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:21.968857 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:21.968979 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:21.973605 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:46:21.973611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:46:21.973615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:46:21.973663 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:46:21.976708 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:21.977645 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:21.977651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:21.977654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:21.977804 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:21.978296 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:21.978300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:21.978303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:21.978618 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:46:21.978737 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:21.981912 (+ 3175us) server_negotiation.cc:299] Negotiation successful
0928 10:46:21.982161 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19008,"thread_start_us":19183,"threads_started":1}
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:46:21.984540 20621 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44205
I20260928 10:46:21.984876 20621 heartbeater.cc:464] Registering TS with master...
I20260928 10:46:21.985664 20621 heartbeater.cc:511] Master 127.8.218.254:44205 requested a full tablet report, sending...
I20260928 10:46:21.987795 20408 ts_manager.cc:194] Registered new tserver with Master: 14741664038a4e48aedccf863b39a7d9 (127.8.218.193:33923)
I20260928 10:46:21.989274 20408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:56051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
W20260928 10:46:22.044745 9067 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.218.194: real 0.135s user 0.003s sys 0.013s
I20260928 10:46:22.045684 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:34219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44205
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:46:22 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592381, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
W20260928 10:46:22.496771 20628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:46:22.497087 20628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:46:22.497233 20628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:46:22.504701 20628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:46:22.504841 20628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:46:22.504992 20628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:46:22.515282 20628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.218.212:34219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:46:22.516646 20628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:46:22.518024 20628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:46:22.531530 20641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:46:22.531710 20642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:46:22.532109 20644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:46:22.532804 20628 server_base.cc:1034] running on GCE node
I20260928 10:46:22.533488 20628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:46:22.534523 20628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:46:22.535789 20628 hybrid_clock.cc:648] HybridClock initialized: now 1790592382535728 us; error 47 us; skew 500 ppm
I20260928 10:46:22.536163 20628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:46:22 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:46:22.539790 20628 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:46:22.542606 20628 webserver.cc:460] Webserver started at http://127.8.218.194:43745/ using document root <none> and password file <none>
I20260928 10:46:22.543296 20628 fs_manager.cc:359] Metadata directory not provided
I20260928 10:46:22.543402 20628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:46:22.543711 20628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:46:22 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:46:22.560920 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b"
format_stamp: "Formatted at 2026-09-28 10:46:22 on dist-test-slave-mn38"
server_key: "c28b685353c7fdeb61a501a6d486d54a"
server_key_iv: "ecd1620febb87b5f71ea02c217786b89"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:22.562150 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b"
format_stamp: "Formatted at 2026-09-28 10:46:22 on dist-test-slave-mn38"
server_key: "c28b685353c7fdeb61a501a6d486d54a"
server_key_iv: "ecd1620febb87b5f71ea02c217786b89"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:22.572247 20628 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s
I20260928 10:46:22.590219 20651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:46:22.592332 20628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260928 10:46:22.592621 20628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b"
format_stamp: "Formatted at 2026-09-28 10:46:22 on dist-test-slave-mn38"
server_key: "c28b685353c7fdeb61a501a6d486d54a"
server_key_iv: "ecd1620febb87b5f71ea02c217786b89"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:22.592979 20628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:46:22.634797 20628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:46:22.640301 20628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:46:22.640723 20628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:46:22.642059 20628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:46:22.644025 20628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:46:22.644172 20628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:46:22.644361 20628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:46:22.644470 20628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:46:22.690971 20628 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:43761
I20260928 10:46:22.691000 20763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:43761 every 8 connection(s)
I20260928 10:46:22.692796 20628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:46:22.699141 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 20628
I20260928 10:46:22.699597 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:46:22.700168 9067 external_mini_cluster.cc:1428] Setting key e8a1427979edd7c14b8f2b8cfeacff60
Sep 28 10:46:22 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:46:22.732232 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:22.696852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:54979 (local address 127.8.218.254:44205)
0928 10:46:22.712170 (+ 15318us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:22.712177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:22.712203 (+ 26us) server_negotiation.cc:407] Connection header received
0928 10:46:22.712273 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:22.712278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:22.712364 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:22.712513 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:46:22.714879 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:22.715931 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:22.719777 (+ 3846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:22.720349 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:22.720451 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:22.724926 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:46:22.724932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:46:22.724937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:46:22.724980 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:46:22.728710 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:22.729545 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:22.729552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:22.729555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:22.729691 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:22.730140 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:22.730144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:22.730146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:22.730457 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:46:22.730596 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:22.731694 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0928 10:46:22.731893 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15170,"thread_start_us":15798,"threads_started":1}
I20260928 10:46:22.733590 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:22.696228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44205 (local address 127.8.218.194:54979)
0928 10:46:22.702156 (+ 5928us) negotiation.cc:107] Waiting for socket to connect
0928 10:46:22.702217 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 10:46:22.704890 (+ 2673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:46:22.712874 (+ 7984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:46:22.712890 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:46:22.713629 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:46:22.714655 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:22.714688 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:22.717609 (+ 2921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:22.717615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:22.719573 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:22.719593 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:22.720812 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:22.720818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:22.721115 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:22.721961 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:46:22.721991 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:46:22.724646 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:46:22.728878 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:46:22.728889 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:46:22.728909 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:46:22.729370 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:46:22.729807 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:46:22.729812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:46:22.729814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:46:22.730009 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:46:22.730716 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:46:22.730727 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:46:22.731216 (+ 489us) client_negotiation.cc:769] Sending connection context
0928 10:46:22.731553 (+ 337us) client_negotiation.cc:240] Negotiation successful
0928 10:46:22.732134 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5532,"thread_start_us":11436,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
I20260928 10:46:22.735212 20764 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44205
I20260928 10:46:22.735569 20764 heartbeater.cc:464] Registering TS with master...
I20260928 10:46:22.736407 20764 heartbeater.cc:511] Master 127.8.218.254:44205 requested a full tablet report, sending...
I20260928 10:46:22.738162 20408 ts_manager.cc:194] Registered new tserver with Master: aed0b7257c8a40c2b8240b6ae5d6d23b (127.8.218.194:43761)
I20260928 10:46:22.739339 20408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:54979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:46:22.760865 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:34219/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44205
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:46:22.992587 20621 heartbeater.cc:503] Master 127.8.218.254:44205 was elected leader, sending a full tablet report...
W20260928 10:46:23.108748 20770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:46:23.109053 20770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:46:23.109184 20770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:46:23.116866 20770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:46:23.116997 20770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:46:23.117146 20770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:46:23.127331 20770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.218.212:34219/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:46:23.128656 20770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:46:23.130048 20770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:46:23.143688 20776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:46:23.143806 20775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:46:23.143798 20778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:46:23.144913 20770 server_base.cc:1034] running on GCE node
I20260928 10:46:23.145933 20770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:46:23.147040 20770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:46:23.148274 20770 hybrid_clock.cc:648] HybridClock initialized: now 1790592383148203 us; error 61 us; skew 500 ppm
I20260928 10:46:23.148681 20770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:46:23 dist-test-slave-mn38 krb5kdc[19932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:46:23.152338 20770 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:46:23.153941 20770 webserver.cc:460] Webserver started at http://127.8.218.195:39705/ using document root <none> and password file <none>
I20260928 10:46:23.154670 20770 fs_manager.cc:359] Metadata directory not provided
I20260928 10:46:23.154784 20770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:46:23.155110 20770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:46:23 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:46:23.170397 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "c2aaceba8d8e451389f5f65e969038bb"
format_stamp: "Formatted at 2026-09-28 10:46:23 on dist-test-slave-mn38"
server_key: "366402f1264287f4eb0c4a4116a1a43a"
server_key_iv: "f6b9040d846d4f8c644ea3ebdcc2b984"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:23.171280 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "c2aaceba8d8e451389f5f65e969038bb"
format_stamp: "Formatted at 2026-09-28 10:46:23 on dist-test-slave-mn38"
server_key: "366402f1264287f4eb0c4a4116a1a43a"
server_key_iv: "f6b9040d846d4f8c644ea3ebdcc2b984"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:23.176977 20770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 10:46:23.193216 20785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:46:23.195210 20770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:46:23.195416 20770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "c2aaceba8d8e451389f5f65e969038bb"
format_stamp: "Formatted at 2026-09-28 10:46:23 on dist-test-slave-mn38"
server_key: "366402f1264287f4eb0c4a4116a1a43a"
server_key_iv: "f6b9040d846d4f8c644ea3ebdcc2b984"
server_key_version: "kuduclusterkey@0"
I20260928 10:46:23.195684 20770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:46:23.213893 20770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:46:23.218446 20770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:46:23.218714 20770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:46:23.219700 20770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:46:23.221055 20770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:46:23.221125 20770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:46:23.221220 20770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:46:23.221262 20770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:46:23.255072 20770 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:42709
I20260928 10:46:23.255095 20897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:42709 every 8 connection(s)
I20260928 10:46:23.256345 20770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:46:23.265965 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 20770
I20260928 10:46:23.266139 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:46:23.266533 9067 external_mini_cluster.cc:1428] Setting key 1c4e28db0c68addec126606b3c8b8e10
Sep 28 10:46:23 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:46:23.277163 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.259319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:57311 (local address 127.8.218.254:44205)
0928 10:46:23.259844 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.259849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.260750 (+ 901us) server_negotiation.cc:407] Connection header received
0928 10:46:23.262526 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.262532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.262617 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.262724 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:46:23.264177 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.265036 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.266732 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.267326 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.267455 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.271280 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:46:23.271285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:46:23.271289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:46:23.271323 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:46:23.273981 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.274853 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.274858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.274860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.275003 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.275464 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.275472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.275474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.275757 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:46:23.275890 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.276695 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.276874 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":225,"threads_started":1}
I20260928 10:46:23.278122 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.259840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44205 (local address 127.8.218.195:57311)
0928 10:46:23.260540 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 10:46:23.260584 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:46:23.262233 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:46:23.262966 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:46:23.262975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:46:23.263413 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:46:23.264002 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.264019 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.265201 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.265205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.266564 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.266578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.267594 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.267602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.267868 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.268646 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:46:23.268671 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:46:23.271029 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:46:23.274180 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:46:23.274188 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:46:23.274204 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:46:23.274699 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:46:23.275131 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:46:23.275134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:46:23.275137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:46:23.275344 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:46:23.275906 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:46:23.275913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:46:23.276304 (+ 391us) client_negotiation.cc:769] Sending connection context
0928 10:46:23.276554 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 10:46:23.276954 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":282,"threads_started":1}
I20260928 10:46:23.279584 20898 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44205
I20260928 10:46:23.279903 20898 heartbeater.cc:464] Registering TS with master...
I20260928 10:46:23.281134 20898 heartbeater.cc:511] Master 127.8.218.254:44205 requested a full tablet report, sending...
I20260928 10:46:23.282737 20407 ts_manager.cc:194] Registered new tserver with Master: c2aaceba8d8e451389f5f65e969038bb (127.8.218.195:42709)
I20260928 10:46:23.283607 20407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:57311
I20260928 10:46:23.295140 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:46:23.312238 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.299529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.8.218.254:44205)
0928 10:46:23.299708 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.299714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.300224 (+ 510us) server_negotiation.cc:407] Connection header received
0928 10:46:23.300343 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.300348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.300441 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.300538 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:46:23.302072 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.302898 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.304181 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.304612 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.304766 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.306891 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:46:23.306895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:46:23.306899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:46:23.306930 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:46:23.309678 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.310379 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.310383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.310386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.310555 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.310972 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.310976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.310979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.311285 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:46:23.311471 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.311814 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.311992 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:46:23.316495 20407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:46:23.324056 20407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:46:23.348984 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.337191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.8.218.193:33923)
0928 10:46:23.337882 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.337889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.339012 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 10:46:23.339528 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.339536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.339863 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.340035 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:46:23.341279 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.342826 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.346180 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.347671 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.347797 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.348041 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.348452 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.348629 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":298,"threads_started":1}
I20260928 10:46:23.349156 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.339811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:42709 (local address 127.0.0.1:41030)
0928 10:46:23.340279 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 10:46:23.340300 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:46:23.340405 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:46:23.341661 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:46:23.341665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:46:23.341691 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:46:23.341978 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.341989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.344017 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.344021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.345791 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.345804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.347913 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.347919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.348142 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.348249 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 10:46:23.348386 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 10:46:23.348668 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":92,"spinlock_wait_cycles":3840,"thread_start_us":613,"threads_started":1}
I20260928 10:46:23.349176 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.336866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:33923 (local address 127.0.0.1:60740)
0928 10:46:23.338858 (+ 1992us) negotiation.cc:107] Waiting for socket to connect
0928 10:46:23.338898 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:46:23.339253 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:46:23.340350 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:46:23.340357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:46:23.340418 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:46:23.341106 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.341120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.342975 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.342979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.345998 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.346014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.347997 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.348002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.348224 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.348249 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:46:23.348386 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 10:46:23.348668 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1701,"mutex_wait_us":306,"spinlock_wait_cycles":22272,"thread_start_us":2203,"threads_started":1}
I20260928 10:46:23.350389 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.338047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:43761 (local address 127.0.0.1:46960)
0928 10:46:23.339391 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0928 10:46:23.339408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:46:23.339527 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:46:23.340363 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:46:23.340367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:46:23.340423 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:46:23.340917 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.340930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.342644 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.342650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.345998 (+ 3348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.346014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.349761 (+ 3747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.349768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.349919 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.349942 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 10:46:23.350011 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 10:46:23.350104 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"mutex_wait_us":217,"spinlock_wait_cycles":36224,"thread_start_us":376,"threads_started":1}
I20260928 10:46:23.350606 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.338528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.8.218.194:43761)
0928 10:46:23.339081 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.339089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.339674 (+ 585us) server_negotiation.cc:407] Connection header received
0928 10:46:23.339761 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.339768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.339965 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.340146 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:46:23.341156 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.342509 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.348152 (+ 5643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.349516 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.349661 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.349857 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.350076 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.350231 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":279,"threads_started":1}
I20260928 10:46:23.351305 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.340292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.8.218.195:42709)
0928 10:46:23.341079 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.341087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.341116 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:46:23.341194 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.341201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.341419 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.341651 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:46:23.342158 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.343871 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.345957 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.347670 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.347829 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.348041 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.349699 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.350978 (+ 1279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":409,"threads_started":1}
I20260928 10:46:23.353780 20699 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3a05bf5f924ea395bdba23c4f3ba53 (DEFAULT_TABLE table=test-table [id=980995b4d56647e18143059209a72a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:46:23.353798 20556 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3a05bf5f924ea395bdba23c4f3ba53 (DEFAULT_TABLE table=test-table [id=980995b4d56647e18143059209a72a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:46:23.355036 20699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3a05bf5f924ea395bdba23c4f3ba53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:46:23.355214 20556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3a05bf5f924ea395bdba23c4f3ba53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:46:23.354882 20833 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3a05bf5f924ea395bdba23c4f3ba53 (DEFAULT_TABLE table=test-table [id=980995b4d56647e18143059209a72a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:46:23.356159 20833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3a05bf5f924ea395bdba23c4f3ba53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:46:23.367309 20917 tablet_bootstrap.cc:492] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Bootstrap starting.
I20260928 10:46:23.369598 20918 tablet_bootstrap.cc:492] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Bootstrap starting.
I20260928 10:46:23.370498 20919 tablet_bootstrap.cc:492] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: Bootstrap starting.
I20260928 10:46:23.372021 20917 tablet_bootstrap.cc:654] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Neither blocks nor log segments found. Creating new log.
I20260928 10:46:23.373926 20917 log.cc:824] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:46:23.374377 20919 tablet_bootstrap.cc:654] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: Neither blocks nor log segments found. Creating new log.
I20260928 10:46:23.375625 20919 log.cc:824] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: Log is configured to *not* fsync() on all Append() calls
I20260928 10:46:23.377095 20917 tablet_bootstrap.cc:492] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: No bootstrap required, opened a new log
I20260928 10:46:23.377357 20917 ts_tablet_manager.cc:1397] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260928 10:46:23.378213 20918 tablet_bootstrap.cc:654] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Neither blocks nor log segments found. Creating new log.
I20260928 10:46:23.378832 20919 tablet_bootstrap.cc:492] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: No bootstrap required, opened a new log
I20260928 10:46:23.379117 20919 ts_tablet_manager.cc:1397] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 10:46:23.379459 20918 log.cc:824] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Log is configured to *not* fsync() on all Append() calls
I20260928 10:46:23.385128 20917 raft_consensus.cc:348] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.385529 20917 raft_consensus.cc:374] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:46:23.385643 20917 raft_consensus.cc:729] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed0b7257c8a40c2b8240b6ae5d6d23b, State: Initialized, Role: FOLLOWER
I20260928 10:46:23.386479 20917 consensus_queue.cc:260] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.387583 20764 heartbeater.cc:503] Master 127.8.218.254:44205 was elected leader, sending a full tablet report...
I20260928 10:46:23.388094 20917 ts_tablet_manager.cc:1428] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260928 10:46:23.388579 20918 tablet_bootstrap.cc:492] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: No bootstrap required, opened a new log
I20260928 10:46:23.388895 20918 ts_tablet_manager.cc:1397] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.003s
I20260928 10:46:23.391690 20919 raft_consensus.cc:348] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.392102 20919 raft_consensus.cc:374] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:46:23.392195 20919 raft_consensus.cc:729] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2aaceba8d8e451389f5f65e969038bb, State: Initialized, Role: FOLLOWER
I20260928 10:46:23.392773 20919 consensus_queue.cc:260] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.393927 20898 heartbeater.cc:503] Master 127.8.218.254:44205 was elected leader, sending a full tablet report...
I20260928 10:46:23.394618 20919 ts_tablet_manager.cc:1428] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 10:46:23.396859 20923 raft_consensus.cc:482] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:46:23.397114 20923 raft_consensus.cc:504] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.398277 20918 raft_consensus.cc:348] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.398749 20918 raft_consensus.cc:374] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:46:23.398857 20918 raft_consensus.cc:729] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14741664038a4e48aedccf863b39a7d9, State: Initialized, Role: FOLLOWER
I20260928 10:46:23.399056 20923 leader_election.cc:288] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14741664038a4e48aedccf863b39a7d9 (127.8.218.193:33923), c2aaceba8d8e451389f5f65e969038bb (127.8.218.195:42709)
I20260928 10:46:23.399513 20918 consensus_queue.cc:260] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.401882 20918 ts_tablet_manager.cc:1428] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260928 10:46:23.407522 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.399412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:57737 (local address 127.8.218.193:33923)
0928 10:46:23.399631 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.399638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.400827 (+ 1189us) server_negotiation.cc:407] Connection header received
0928 10:46:23.401133 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.401139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.401216 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.401283 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:46:23.402704 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.403710 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.405864 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.406664 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.406811 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.406887 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.407170 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.407295 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:46:23.407596 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.399386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:33923 (local address 127.8.218.194:57737)
0928 10:46:23.400724 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0928 10:46:23.400748 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:46:23.400951 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:46:23.401556 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:46:23.401561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:46:23.401588 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:46:23.402543 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.402558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.403857 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.403862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.405733 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.405751 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.406863 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.406869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.407019 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.407043 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 10:46:23.407127 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 10:46:23.407260 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":341,"threads_started":1}
I20260928 10:46:23.408803 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.401353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:59669 (local address 127.8.218.195:42709)
0928 10:46:23.401544 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.401550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.402708 (+ 1158us) server_negotiation.cc:407] Connection header received
0928 10:46:23.402958 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.402964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.403046 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.403152 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:46:23.403721 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.404748 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.406799 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.407587 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.407817 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.407881 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.408436 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.408571 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 10:46:23.408670 20576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3a05bf5f924ea395bdba23c4f3ba53" candidate_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14741664038a4e48aedccf863b39a7d9" is_pre_election: true
I20260928 10:46:23.408890 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.400938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:42709 (local address 127.8.218.194:59669)
0928 10:46:23.402597 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0928 10:46:23.402612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:46:23.402804 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:46:23.403165 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:46:23.403169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:46:23.403194 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:46:23.403528 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.403539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.404911 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.404916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.406643 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:46:23.406655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.408099 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.408106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:46:23.408276 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.408307 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:46:23.408388 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 10:46:23.408510 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"thread_start_us":1344,"threads_started":1}
I20260928 10:46:23.409209 20576 raft_consensus.cc:2454] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aed0b7257c8a40c2b8240b6ae5d6d23b in term 0.
I20260928 10:46:23.409910 20654 leader_election.cc:302] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14741664038a4e48aedccf863b39a7d9, aed0b7257c8a40c2b8240b6ae5d6d23b; no voters:
I20260928 10:46:23.410262 20923 raft_consensus.cc:2781] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:46:23.410378 20923 raft_consensus.cc:482] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:46:23.410239 20853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3a05bf5f924ea395bdba23c4f3ba53" candidate_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2aaceba8d8e451389f5f65e969038bb" is_pre_election: true
I20260928 10:46:23.410470 20923 raft_consensus.cc:3037] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:46:23.410677 20853 raft_consensus.cc:2454] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aed0b7257c8a40c2b8240b6ae5d6d23b in term 0.
I20260928 10:46:23.412786 20923 raft_consensus.cc:504] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.413405 20923 leader_election.cc:288] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [CANDIDATE]: Term 1 election: Requested vote from peers 14741664038a4e48aedccf863b39a7d9 (127.8.218.193:33923), c2aaceba8d8e451389f5f65e969038bb (127.8.218.195:42709)
I20260928 10:46:23.413921 20576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3a05bf5f924ea395bdba23c4f3ba53" candidate_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14741664038a4e48aedccf863b39a7d9"
I20260928 10:46:23.413936 20853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3a05bf5f924ea395bdba23c4f3ba53" candidate_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2aaceba8d8e451389f5f65e969038bb"
I20260928 10:46:23.414141 20853 raft_consensus.cc:3037] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:46:23.414142 20576 raft_consensus.cc:3037] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:46:23.417250 20576 raft_consensus.cc:2454] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aed0b7257c8a40c2b8240b6ae5d6d23b in term 1.
I20260928 10:46:23.417249 20853 raft_consensus.cc:2454] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aed0b7257c8a40c2b8240b6ae5d6d23b in term 1.
I20260928 10:46:23.417717 20654 leader_election.cc:302] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aed0b7257c8a40c2b8240b6ae5d6d23b, c2aaceba8d8e451389f5f65e969038bb; no voters:
I20260928 10:46:23.418030 20923 raft_consensus.cc:2781] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:46:23.418452 20923 raft_consensus.cc:686] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 1 LEADER]: Becoming Leader. State: Replica: aed0b7257c8a40c2b8240b6ae5d6d23b, State: Running, Role: LEADER
I20260928 10:46:23.418912 20923 consensus_queue.cc:237] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } }
I20260928 10:46:23.423727 20407 catalog_manager.cc:5495] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b reported cstate change: term changed from 0 to 1, leader changed from <none> to aed0b7257c8a40c2b8240b6ae5d6d23b (127.8.218.194). New cstate: current_term: 1 leader_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed0b7257c8a40c2b8240b6ae5d6d23b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 43761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 } health_report { overall_health: UNKNOWN } } }
I20260928 10:46:23.452731 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.447482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.8.218.194:43761)
0928 10:46:23.447688 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.447693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.447712 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:46:23.447895 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.447899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.447975 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.448113 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:46:23.448475 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.449183 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.450583 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.450979 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.451094 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.451594 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:46:23.452118 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:46:23.452253 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.452334 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.452412 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:46:23.460953 20853 raft_consensus.cc:1261] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 1 FOLLOWER]: Refusing update from remote peer aed0b7257c8a40c2b8240b6ae5d6d23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:46:23.461006 20576 raft_consensus.cc:1261] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 1 FOLLOWER]: Refusing update from remote peer aed0b7257c8a40c2b8240b6ae5d6d23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:46:23.461858 20923 consensus_queue.cc:1035] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2aaceba8d8e451389f5f65e969038bb" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:46:23.462204 20928 consensus_queue.cc:1035] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14741664038a4e48aedccf863b39a7d9" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:46:23.463977 20765 tablet.cc:2369] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:46:23.465893 20623 tablet.cc:2369] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:46:23.489296 20408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:46:23.489603 20408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:46:23.493382 20408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e88c7d6ede40549403a305763aba18: Sending DeleteTablet for 3 replicas of tablet 8e3a05bf5f924ea395bdba23c4f3ba53
I20260928 10:46:23.494720 20699 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3a05bf5f924ea395bdba23c4f3ba53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:46:23 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:46960
I20260928 10:46:23.494766 20556 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3a05bf5f924ea395bdba23c4f3ba53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:46:23 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:60740
I20260928 10:46:23.494752 20833 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3a05bf5f924ea395bdba23c4f3ba53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:46:23 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:41030
I20260928 10:46:23.496236 20942 tablet_replica.cc:330] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: stopping tablet replica
I20260928 10:46:23.496740 20942 raft_consensus.cc:2229] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:46:23.497190 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 20488
I20260928 10:46:23.497395 20942 raft_consensus.cc:2258] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:46:23.499832 20942 ts_tablet_manager.cc:1905] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:46:23.507470 20942 ts_tablet_manager.cc:1918] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:46:23.507793 20942 log.cc:1196] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/8e3a05bf5f924ea395bdba23c4f3ba53
I20260928 10:46:23.496037 20941 tablet_replica.cc:330] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: stopping tablet replica
I20260928 10:46:23.508481 20942 ts_tablet_manager.cc:1939] T 8e3a05bf5f924ea395bdba23c4f3ba53 P c2aaceba8d8e451389f5f65e969038bb: Deleting consensus metadata
I20260928 10:46:23.508723 20941 raft_consensus.cc:2229] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:46:23.509352 20941 raft_consensus.cc:2258] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:46:23.510637 20394 catalog_manager.cc:4841] TS c2aaceba8d8e451389f5f65e969038bb (127.8.218.195:42709): tablet 8e3a05bf5f924ea395bdba23c4f3ba53 (table test-table [id=980995b4d56647e18143059209a72a0a]) successfully deleted
I20260928 10:46:23.512097 20941 ts_tablet_manager.cc:1905] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:46:23.512305 20943 tablet_replica.cc:330] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: stopping tablet replica
I20260928 10:46:23.512717 20943 raft_consensus.cc:2229] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:46:23.513460 20943 raft_consensus.cc:2258] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:46:23.516995 20943 ts_tablet_manager.cc:1905] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:46:23.519445 20941 ts_tablet_manager.cc:1918] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:46:23.519650 20941 log.cc:1196] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/8e3a05bf5f924ea395bdba23c4f3ba53
I20260928 10:46:23.520200 20941 ts_tablet_manager.cc:1939] T 8e3a05bf5f924ea395bdba23c4f3ba53 P 14741664038a4e48aedccf863b39a7d9: Deleting consensus metadata
Sep 28 10:46:23 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:46:23.522233 20394 catalog_manager.cc:4841] TS 14741664038a4e48aedccf863b39a7d9 (127.8.218.193:33923): tablet 8e3a05bf5f924ea395bdba23c4f3ba53 (table test-table [id=980995b4d56647e18143059209a72a0a]) successfully deleted
I20260928 10:46:23.525393 20943 ts_tablet_manager.cc:1918] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:46:23.525612 20943 log.cc:1196] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/8e3a05bf5f924ea395bdba23c4f3ba53
I20260928 10:46:23.526149 20943 ts_tablet_manager.cc:1939] T 8e3a05bf5f924ea395bdba23c4f3ba53 P aed0b7257c8a40c2b8240b6ae5d6d23b: Deleting consensus metadata
I20260928 10:46:23.527937 20395 catalog_manager.cc:4841] TS aed0b7257c8a40c2b8240b6ae5d6d23b (127.8.218.194:43761): tablet 8e3a05bf5f924ea395bdba23c4f3ba53 (table test-table [id=980995b4d56647e18143059209a72a0a]) successfully deleted
I20260928 10:46:23.530169 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.498797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.8.218.193:33923)
0928 10:46:23.498973 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.498980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.508695 (+ 9715us) server_negotiation.cc:407] Connection header received
0928 10:46:23.509228 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.509234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.509319 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.509400 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:46:23.511811 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.512866 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.515087 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.515712 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.515847 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.522098 (+ 6251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:46:23.522115 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:46:23.522134 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:46:23.522189 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:46:23.526698 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.527764 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.527775 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.527779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.527963 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.528475 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.528479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.528481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.528877 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:46:23.529133 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.529614 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.529799 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:46:23.530786 20616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:46:23.643071 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 20488
I20260928 10:46:23.657410 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 20628
Sep 28 10:46:23 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:46:23.674808 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.658193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.8.218.194:43761)
0928 10:46:23.658362 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.658366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.658455 (+ 89us) server_negotiation.cc:407] Connection header received
0928 10:46:23.658673 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.658676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.658743 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.658829 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:46:23.660419 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.661242 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.663128 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.663552 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.663653 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.668847 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:46:23.668863 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:46:23.668878 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:46:23.668916 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:46:23.672018 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.672708 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.672715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.672718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.672873 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.673267 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.673271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.673273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.673664 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:46:23.673992 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.674233 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.674437 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:46:23.675595 20759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:46:23.787461 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 20628
I20260928 10:46:23.802083 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 20770
Sep 28 10:46:23 dist-test-slave-mn38 krb5kdc[19932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:46:23.817212 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:46:23.802812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.8.218.195:42709)
0928 10:46:23.802981 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 10:46:23.802987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:46:23.803078 (+ 91us) server_negotiation.cc:407] Connection header received
0928 10:46:23.803270 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:46:23.803277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:46:23.803357 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:46:23.803441 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:46:23.804733 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.805520 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.806792 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:46:23.807186 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:46:23.807305 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:46:23.811592 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:46:23.811607 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:46:23.811624 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:46:23.811686 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:46:23.814498 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.815230 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.815238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.815241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.815374 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:46:23.815745 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:46:23.815752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:46:23.815755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:46:23.816182 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:46:23.816402 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 10:46:23.816759 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 10:46:23.816939 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:46:23.817934 20893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:46:23.905036 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 20770
I20260928 10:46:23.917784 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 20376
I20260928 10:46:23.918911 20438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:46:24.028414 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 20376
I20260928 10:46:24.051312 9067 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 10:46:24.387060 9067 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 10:46:24.387295 9067 mini_ranger.cc:66] Stopping Ranger...
I20260928 10:46:24.723091 9067 mini_ranger.cc:68] Stopped Ranger
2026-09-28 10:46:24.723 UTC [19962] LOG: received smart shutdown request
2026-09-28 10:46:24.726 UTC [19962] LOG: background worker "logical replication launcher" (PID 19971) exited with exit code 1
2026-09-28 10:46:24.726 UTC [19966] LOG: shutting down
2026-09-28 10:46:24.899 UTC [19962] LOG: database system is shut down
2026-09-28T10:46:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50535 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:46:25 dist-test-slave-mn38 krb5kdc[20957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:46:25 dist-test-slave-mn38 krb5kdc[20957](info): set up 2 sockets
Sep 28 10:46:25 dist-test-slave-mn38 krb5kdc[20957](info): commencing operation
krb5kdc: starting...
W20260928 10:46:26.026767 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:46:26 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:46:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.218.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.218.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/rangeradmin_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/rangeradmin_127.8.218.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.218.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.218.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/rangerlookup_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/rangerlookup_127.8.218.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.218.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.218.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/HTTP_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/HTTP_127.8.218.212@KRBTEST.COM.keytab.
I20260928 10:46:26.340941 9067 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3zclSv/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/postgres -l logfile start
2026-09-28 10:46:28.989 UTC [20987] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 10:46:28.990 UTC [20987] LOG: listening on IPv4 address "127.8.218.212", port 37819
2026-09-28 10:46:28.994 UTC [20987] LOG: listening on Unix socket "/tmp/.s.PGSQL.37819"
2026-09-28 10:46:29.015 UTC [20990] LOG: database system was shut down at 2026-09-28 10:46:27 UTC
2026-09-28 10:46:29.018 UTC [20987] LOG: database system is ready to accept connections
I20260928 10:46:29.910408 9067 mini_postgres.cc:96] Postgres bound to 37819
2026-09-28 10:46:29.930 UTC [20998] FATAL: database "slave" does not exist
127.8.218.212:37819 - accepting connections
I20260928 10:46:29.931385 9067 mini_ranger.cc:161] Starting Ranger...
I20260928 10:46:29.956923 9067 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 10:46:30.485140 9067 mini_ranger.cc:87] Created ranger Postgres database
I20260928 10:46:30.485237 9067 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-admin
2026-09-28 10:46:30,772 [I] DB FLAVOR :POSTGRES
2026-09-28 10:46:30,773 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:46:30,773 [I] Checking connection..
2026-09-28 10:46:30,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 10:46:31,042 [I] Checking connection passed.
2026-09-28 10:46:31,042 [I] --------- Verifying version history table ---------
2026-09-28 10:46:31,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:46:31,307 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:46:31,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:46:31,579 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 10:46:31,579 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 10:46:31,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 10:46:31.837 UTC [21062] WARNING: there is no transaction in progress
2026-09-28 10:46:31,843 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 10:46:31,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 10:46:32,107 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 10:46:32,107 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:46:32,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 10:46:32,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 10:46:32,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mn38.c.gcp-upstream.internal','N') ;"
2026-09-28 10:46:32,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:46:33,111 [I] Table x_portal_user does not exist in database ranger
2026-09-28 10:46:33,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:46:33,349 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 10:46:33,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:46:33,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:46:33,848 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 10:46:33,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 10:46:34.868 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:34.887 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:34.906 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:34.925 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:34.942 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:34.961 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.025 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.034 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.051 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.062 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.074 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.090 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.101 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.109 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.117 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35.125 UTC [21170] WARNING: there is no transaction in progress
2026-09-28 10:46:35,873 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 10:46:35,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 10:46:36,115 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 10:46:36,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 10:46:36,382 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 10:46:36,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 10:46:36,630 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 10:46:36,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 10:46:36,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 10:46:37,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mn38.c.gcp-upstream.internal';"
2026-09-28 10:46:37,341 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 10:46:37,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 10:46:37,577 [I] DB_PATCHES have already been applied
I20260928 10:46:37.583627 9067 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-admin:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3zclSv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 10:46:37.583717 9067 mini_ranger.cc:193] Using host: 127.8.218.212
I20260928 10:46:37.598791 9067 mini_ranger.cc:227] Ranger admin URL: http://127.8.218.212:48171
Sep 28, 2026 10:46:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:46:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:46:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:46:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:46:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:46:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:46:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 10:46:38.647830 9067 mini_ranger.cc:160] Time spent starting Ranger: real 8.716s user 0.000s sys 0.040s
Sep 28, 2026 10:46:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:46:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 10:46:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:46:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:47:00 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 10:47:01.312242 9067 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.218.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.218.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/rangerkms_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.218.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/rangerkms_127.8.218.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.218.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.218.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.218.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/HTTP_127.8.218.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.218.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/HTTP_127.8.218.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 10:47:01 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592421, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 10:47:01.502496 9067 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 10:47:01.522795 9067 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 10:47:02.591857 9067 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 10:47:02.591967 9067 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms
I20260928 10:47:02.592007 9067 mini_ranger_kms.cc:227] Using postgres at 127.8.218.212:37819
2026-09-28 10:47:02,927 [I] DB FLAVOR :POSTGRES
2026-09-28 10:47:02,928 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 10:47:02,928 [I] Checking connection
2026-09-28 10:47:02,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 10:47:03,177 [I] connection success
2026-09-28 10:47:03,178 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 10:47:03,178 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 10:47:03,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 10:47:03,417 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 10:47:03,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 10:47:03,418 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 10:47:03,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.218.212:37819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 10:47:03,692 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 10:47:04.212287 9067 mini_ranger_kms.cc:310] Created kms service
I20260928 10:47:04.415316 9067 mini_ranger_kms.cc:326] Created kudu user
I20260928 10:47:04.467599 9067 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 10:47:04 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592421, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:04.929616 9067 mini_ranger_kms.cc:384] Added ranger policy
I20260928 10:47:04.929893 9067 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms:/tmp/dist-test-task3zclSv/build/asan/bin/postgresql.jar:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3zclSv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 10:47:04.929931 9067 mini_ranger_kms.cc:240] Using host: 127.8.218.212
I20260928 10:47:04.947074 9067 mini_ranger_kms.cc:276] Ranger KMS PID: 21351
I20260928 10:47:04.947193 9067 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.218.212:40207
Sep 28, 2026 10:47:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:47:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:47:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:47:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 10:47:06 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592426, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.218.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:07.042939 9067 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.540s user 0.072s sys 0.274s
I20260928 10:47:07.043180 9067 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 10:47:07.043354 9067 mini_ranger_kms.cc:401] 127.8.218.212:40207/kms/v1/keys
Sep 28 10:47:07 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592426, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.218.212@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
Sep 28 10:47:07 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592426, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.218.212@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
Sep 28 10:47:10 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:47:10.238222 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45579
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:45579
--ranger_config_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:40207/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:10.591747 21401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:10.592108 21401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:10.592249 21401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:10.600243 21401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:10.600412 21401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:10.600493 21401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:10.600571 21401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:10.600641 21401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:10.611285 21401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.218.212:40207/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:45579
--ranger_config_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:10.612705 21401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:10.614179 21401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:10.633062 21406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:10.633091 21407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:10.633136 21409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:10.634622 21401 server_base.cc:1034] running on GCE node
I20260928 10:47:10.635648 21401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:10.637312 21401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:10.638566 21401 hybrid_clock.cc:648] HybridClock initialized: now 1790592430638525 us; error 68 us; skew 500 ppm
I20260928 10:47:10.639002 21401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:10 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:10.645457 21401 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:47:10.647260 21401 webserver.cc:460] Webserver started at http://127.8.218.254:36171/ using document root <none> and password file <none>
I20260928 10:47:10.647969 21401 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:10.648087 21401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:10.648406 21401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:47:10 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:10.703024 21401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "7bf1b9490d25423f966d7812f1064c1f"
format_stamp: "Formatted at 2026-09-28 10:47:10 on dist-test-slave-mn38"
server_key: "4cc51df7f408c3f071ac05df6ede6c3d"
server_key_iv: "2bcfdb54bdeb04b15442de02564f222c"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:10.704186 21401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "7bf1b9490d25423f966d7812f1064c1f"
format_stamp: "Formatted at 2026-09-28 10:47:10 on dist-test-slave-mn38"
server_key: "4cc51df7f408c3f071ac05df6ede6c3d"
server_key_iv: "2bcfdb54bdeb04b15442de02564f222c"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:10.710196 21401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 10:47:10.727691 21416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:10.729504 21401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:47:10.729873 21401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "7bf1b9490d25423f966d7812f1064c1f"
format_stamp: "Formatted at 2026-09-28 10:47:10 on dist-test-slave-mn38"
server_key: "4cc51df7f408c3f071ac05df6ede6c3d"
server_key_iv: "2bcfdb54bdeb04b15442de02564f222c"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:10.730110 21401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:10.760860 21401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:10.765004 21401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:10.765307 21401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:10.784658 21401 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:45579
I20260928 10:47:10.784675 21467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:45579 every 8 connection(s)
I20260928 10:47:10.786250 21401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:47:10.791720 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:10.794821 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 21401
I20260928 10:47:10.795013 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:47:10.795805 9067 external_mini_cluster.cc:1428] Setting key 66ef37ddde22e9da5b862ff544f44617
I20260928 10:47:10.810159 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Bootstrap starting.
I20260928 10:47:10.814273 21468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:10.815464 21468 log.cc:824] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:10.818956 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: No bootstrap required, opened a new log
Sep 28 10:47:10 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:10.828910 21468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } }
I20260928 10:47:10.829481 21468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:10.829594 21468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf1b9490d25423f966d7812f1064c1f, State: Initialized, Role: FOLLOWER
I20260928 10:47:10.830315 21468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } }
I20260928 10:47:10.830592 21468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:47:10.830724 21468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:47:10.830904 21468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:10.834228 21468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } }
I20260928 10:47:10.834456 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:10.798156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.8.218.254:45579)
0928 10:47:10.808769 (+ 10613us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:10.808789 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:10.808853 (+ 64us) server_negotiation.cc:407] Connection header received
0928 10:47:10.811500 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:10.811541 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:10.812239 (+ 698us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:10.812842 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:10.814642 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:10.816721 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:10.818580 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:10.819670 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:10.820649 (+ 979us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:10.825023 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:10.825036 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:10.825060 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:10.825122 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:10.829134 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:10.830199 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:10.830209 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:10.830217 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:10.830419 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:10.830883 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:10.830888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:10.830895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:10.831674 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:10.832014 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:10.832515 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 10:47:10.832980 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10316,"thread_start_us":10054,"threads_started":1}
I20260928 10:47:10.834883 21468 leader_election.cc:302] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf1b9490d25423f966d7812f1064c1f; no voters:
I20260928 10:47:10.835769 21468 leader_election.cc:288] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:47:10.835991 21473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:10.839794 21473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 1 LEADER]: Becoming Leader. State: Replica: 7bf1b9490d25423f966d7812f1064c1f, State: Running, Role: LEADER
I20260928 10:47:10.840344 21473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } }
I20260928 10:47:10.840893 21468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:10.843176 21475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bf1b9490d25423f966d7812f1064c1f. Latest consensus state: current_term: 1 leader_uuid: "7bf1b9490d25423f966d7812f1064c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } } }
I20260928 10:47:10.843149 21474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bf1b9490d25423f966d7812f1064c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } } }
I20260928 10:47:10.843835 21475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:10.843835 21474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:10.848212 21481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:47:10.848791 21468 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 10:47:10.853397 21481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:47:10.875679 21481 catalog_manager.cc:1347] Generated new cluster ID: e114ca88d4c3433284db02882aab39f2
I20260928 10:47:10.875850 21481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:47:10.898808 21481 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:47:10.900244 21481 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:47:10.907891 21481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Generated new TSK 0
I20260928 10:47:10.908845 21481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:47:11.701665 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:40207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45579
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:47:12 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 10:47:12.165534 21513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:12.166066 21513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:12.166242 21513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:12.178326 21513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:12.178473 21513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:12.178725 21513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:47:12.195302 21513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.218.212:40207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:12.197365 21513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:12.199334 21513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:12.221513 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:12.225281 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:12.227584 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:12.228744 21513 server_base.cc:1034] running on GCE node
I20260928 10:47:12.229935 21513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:12.237761 21513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:12.239387 21513 hybrid_clock.cc:648] HybridClock initialized: now 1790592432239265 us; error 125 us; skew 500 ppm
I20260928 10:47:12.239878 21513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:12 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:12.243961 21513 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:47:12.246075 21513 webserver.cc:460] Webserver started at http://127.8.218.193:33257/ using document root <none> and password file <none>
I20260928 10:47:12.246866 21513 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:12.246974 21513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:12.247279 21513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:47:12 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:12.268005 21513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "f33f2af14f114175bb862665fd6d4cac"
format_stamp: "Formatted at 2026-09-28 10:47:12 on dist-test-slave-mn38"
server_key: "0bcca29411923c97186ddf4cb692a65d"
server_key_iv: "fd83a374a8ee8297f6858304d0043ed8"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:12.269420 21513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "f33f2af14f114175bb862665fd6d4cac"
format_stamp: "Formatted at 2026-09-28 10:47:12 on dist-test-slave-mn38"
server_key: "0bcca29411923c97186ddf4cb692a65d"
server_key_iv: "fd83a374a8ee8297f6858304d0043ed8"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:12.294703 21513 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.005s sys 0.011s
I20260928 10:47:12.320979 21533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:12.323176 21513 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.006s
I20260928 10:47:12.323560 21513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "f33f2af14f114175bb862665fd6d4cac"
format_stamp: "Formatted at 2026-09-28 10:47:12 on dist-test-slave-mn38"
server_key: "0bcca29411923c97186ddf4cb692a65d"
server_key_iv: "fd83a374a8ee8297f6858304d0043ed8"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:12.323972 21513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:12.372723 21513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:12.378650 21513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:12.379078 21513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:12.386788 21513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:12.395630 21513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:12.395956 21513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:47:12.396200 21513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:12.396338 21513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 10:47:12 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:12.650838 21513 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:46577
I20260928 10:47:12.650868 21645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:46577 every 8 connection(s)
I20260928 10:47:12.652562 21513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:47:12.660513 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 21513
I20260928 10:47:12.660765 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:47:12.661789 9067 external_mini_cluster.cc:1428] Setting key 21e688be3bb816bd3247f5669cb88c77
Sep 28 10:47:12 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:12.688591 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:12.656826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:50131 (local address 127.8.218.254:45579)
0928 10:47:12.666048 (+ 9222us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:12.666055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:12.669499 (+ 3444us) server_negotiation.cc:407] Connection header received
0928 10:47:12.669580 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:12.669586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:12.669683 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:12.669764 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:12.672060 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:12.673233 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:12.675724 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:12.676366 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:12.676468 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:12.681011 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:12.681019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:12.681024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:12.681071 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:12.684435 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:12.685544 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:12.685561 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:12.685565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:12.685738 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:12.686336 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:12.686342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:12.686344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:12.686721 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:12.686853 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:12.687934 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0928 10:47:12.688197 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9051,"thread_start_us":8913,"threads_started":1}
I20260928 10:47:12.691274 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:12.656201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45579 (local address 127.8.218.193:50131)
0928 10:47:12.666685 (+ 10484us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:12.666754 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:12.669174 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:12.670351 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:12.670364 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:12.670989 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:12.671831 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:12.671863 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:12.673478 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:12.673484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:12.675513 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:12.675531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:12.676845 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:12.676851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:12.677155 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:12.677982 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:12.678013 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:12.680677 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:12.684669 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:12.684679 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:12.684701 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:12.685340 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:12.685910 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:12.685916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:12.685919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:12.686199 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:12.686950 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:12.686959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:12.687450 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 10:47:12.689209 (+ 1759us) client_negotiation.cc:240] Negotiation successful
0928 10:47:12.689835 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10111,"thread_start_us":9994,"threads_started":1}
I20260928 10:47:12.693375 21646 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45579
I20260928 10:47:12.693724 21646 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:12.694528 21646 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
I20260928 10:47:12.697130 21433 ts_manager.cc:194] Registered new tserver with Master: f33f2af14f114175bb862665fd6d4cac (127.8.218.193:46577)
I20260928 10:47:12.699106 21433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:50131
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:47:12.730340 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:40207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45579
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:47:12 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
W20260928 10:47:13.158260 21652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:13.158571 21652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:13.158715 21652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:13.166312 21652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:13.166450 21652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:13.166600 21652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:47:13.176606 21652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.218.212:40207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:13.178076 21652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:13.179437 21652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:13.194451 21667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:13.194451 21666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:13.195294 21669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:13.195789 21652 server_base.cc:1034] running on GCE node
I20260928 10:47:13.196543 21652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:13.197743 21652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:13.198999 21652 hybrid_clock.cc:648] HybridClock initialized: now 1790592433198903 us; error 57 us; skew 500 ppm
I20260928 10:47:13.199415 21652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:13 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:13.203227 21652 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:47:13.204998 21652 webserver.cc:460] Webserver started at http://127.8.218.194:46413/ using document root <none> and password file <none>
I20260928 10:47:13.205884 21652 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:13.206019 21652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:13.206357 21652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:47:13 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:13.224452 21652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "1456460f70f74dee9b4524fa998686b5"
format_stamp: "Formatted at 2026-09-28 10:47:13 on dist-test-slave-mn38"
server_key: "84e6ac3aa281c01790eef31d2d1779e0"
server_key_iv: "f6ee01f2c708332654e9734236b59aec"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:13.225677 21652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "1456460f70f74dee9b4524fa998686b5"
format_stamp: "Formatted at 2026-09-28 10:47:13 on dist-test-slave-mn38"
server_key: "84e6ac3aa281c01790eef31d2d1779e0"
server_key_iv: "f6ee01f2c708332654e9734236b59aec"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:13.232847 21652 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:47:13.251441 21676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:13.253257 21652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 10:47:13.253537 21652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "1456460f70f74dee9b4524fa998686b5"
format_stamp: "Formatted at 2026-09-28 10:47:13 on dist-test-slave-mn38"
server_key: "84e6ac3aa281c01790eef31d2d1779e0"
server_key_iv: "f6ee01f2c708332654e9734236b59aec"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:13.253813 21652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:13.294319 21652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:13.298678 21652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:13.298959 21652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:13.299937 21652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:13.301363 21652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:13.301462 21652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:13.301592 21652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:13.301672 21652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:13.336136 21652 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:33129
I20260928 10:47:13.336272 21788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:33129 every 8 connection(s)
I20260928 10:47:13.337857 21652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:47:13.342154 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 21652
I20260928 10:47:13.342326 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:47:13.342862 9067 external_mini_cluster.cc:1428] Setting key aecc861088abea3dbac4d937073d53ca
Sep 28 10:47:13 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:13.360318 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:13.340695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:32899 (local address 127.8.218.254:45579)
0928 10:47:13.341295 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:13.341302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:13.343548 (+ 2246us) server_negotiation.cc:407] Connection header received
0928 10:47:13.344485 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:13.344492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:13.344578 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:13.344699 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:13.346614 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.347748 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.350124 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.350832 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.350988 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:13.354586 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:13.354592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:13.354595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:13.354630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:13.357079 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:13.357867 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:13.357872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:13.357874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:13.357993 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:13.358465 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:13.358469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:13.358471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:13.358844 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:13.358989 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:13.359734 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 10:47:13.359957 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":306,"threads_started":1}
I20260928 10:47:13.361549 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:13.341291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45579 (local address 127.8.218.194:32899)
0928 10:47:13.342139 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:13.342184 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:13.344131 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:13.344969 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:13.344981 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:13.345607 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:13.346419 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:13.346444 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.347916 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.347921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:13.349936 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:13.349954 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.351108 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.351115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:13.351402 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:13.352216 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:13.352244 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:13.354333 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:13.357220 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:13.357230 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:13.357249 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:13.357742 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:13.358096 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:13.358101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:13.358105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:13.358309 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:13.358960 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:13.358967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:13.359346 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 10:47:13.359597 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 10:47:13.360196 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":368,"threads_started":1}
I20260928 10:47:13.362974 21789 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45579
I20260928 10:47:13.363276 21789 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:13.364071 21789 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
I20260928 10:47:13.365561 21433 ts_manager.cc:194] Registered new tserver with Master: 1456460f70f74dee9b4524fa998686b5 (127.8.218.194:33129)
I20260928 10:47:13.366371 21433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:32899
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:47:13.398902 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:40207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45579
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:47:13.703176 21646 heartbeater.cc:503] Master 127.8.218.254:45579 was elected leader, sending a full tablet report...
W20260928 10:47:13.786275 21795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:13.786641 21795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:13.786841 21795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:13.798056 21795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:13.798257 21795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:13.798444 21795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:47:13.811923 21795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.218.212:40207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:13.813714 21795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:13.815323 21795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:13.829934 21801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:13.830107 21803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:13.830368 21800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:13.831126 21795 server_base.cc:1034] running on GCE node
I20260928 10:47:13.831802 21795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:13.832851 21795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:13.834084 21795 hybrid_clock.cc:648] HybridClock initialized: now 1790592433834057 us; error 53 us; skew 500 ppm
I20260928 10:47:13.834419 21795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:13 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:13.837577 21795 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:47:13.839088 21795 webserver.cc:460] Webserver started at http://127.8.218.195:40417/ using document root <none> and password file <none>
I20260928 10:47:13.839781 21795 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:13.839892 21795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:13.840185 21795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 10:47:13 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:13.857416 21795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "cd5891da25f1459aa63eb0b608f4c2a3"
format_stamp: "Formatted at 2026-09-28 10:47:13 on dist-test-slave-mn38"
server_key: "8a1ed90d85feb1db67862a57d70b7b3c"
server_key_iv: "e54e2e394c3a0b4aa162e8f653f7eeb3"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:13.858340 21795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "cd5891da25f1459aa63eb0b608f4c2a3"
format_stamp: "Formatted at 2026-09-28 10:47:13 on dist-test-slave-mn38"
server_key: "8a1ed90d85feb1db67862a57d70b7b3c"
server_key_iv: "e54e2e394c3a0b4aa162e8f653f7eeb3"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:13.863781 21795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:47:13.877543 21810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:13.879109 21795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:47:13.879280 21795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "cd5891da25f1459aa63eb0b608f4c2a3"
format_stamp: "Formatted at 2026-09-28 10:47:13 on dist-test-slave-mn38"
server_key: "8a1ed90d85feb1db67862a57d70b7b3c"
server_key_iv: "e54e2e394c3a0b4aa162e8f653f7eeb3"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:13.879508 21795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:13.896510 21795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:13.900558 21795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:13.900918 21795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:13.901919 21795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:13.903262 21795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:13.903329 21795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:13.903441 21795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:13.903502 21795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:13.933681 21795 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:42273
I20260928 10:47:13.933719 21922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:42273 every 8 connection(s)
I20260928 10:47:13.935271 21795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:47:13.939868 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 21795
I20260928 10:47:13.940037 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:47:13.940459 9067 external_mini_cluster.cc:1428] Setting key a034f327afd49bf14dac007dfd215116
Sep 28 10:47:13 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:13.955199 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:13.938151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:35683 (local address 127.8.218.254:45579)
0928 10:47:13.938606 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:13.938610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:13.939485 (+ 875us) server_negotiation.cc:407] Connection header received
0928 10:47:13.941371 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:13.941378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:13.941500 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:13.941670 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:13.943284 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.944071 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.945848 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.946270 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.946361 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:13.949643 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:13.949648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:13.949651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:13.949685 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:13.951993 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:13.952773 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:13.952778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:13.952781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:13.952909 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:13.953298 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:13.953301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:13.953303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:13.953649 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:13.953770 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:13.954639 (+ 869us) server_negotiation.cc:299] Negotiation successful
0928 10:47:13.954859 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":219,"threads_started":1}
I20260928 10:47:13.956243 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:13.938646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45579 (local address 127.8.218.195:35683)
0928 10:47:13.939326 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:13.939369 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:13.941071 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:13.941928 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:13.941940 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:13.942504 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:13.943116 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:13.943136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.944212 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.944216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:13.945703 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:13.945719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.946465 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.946470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:13.946720 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:13.947373 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:13.947401 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:13.949393 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:13.952169 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:13.952177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:13.952195 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:13.952658 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:13.952992 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:13.952995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:13.952997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:13.953216 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:13.953775 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:13.953784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:13.954186 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 10:47:13.954488 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 10:47:13.954989 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
I20260928 10:47:13.957647 21923 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45579
I20260928 10:47:13.957966 21923 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:13.958794 21923 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
I20260928 10:47:13.960304 21433 ts_manager.cc:194] Registered new tserver with Master: cd5891da25f1459aa63eb0b608f4c2a3 (127.8.218.195:42273)
I20260928 10:47:13.961129 21433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:35683
I20260928 10:47:13.968713 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:47:13.970381 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 21513
Sep 28 10:47:13 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:47:13.987149 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:13.971336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.8.218.193:46577)
0928 10:47:13.971971 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:13.971978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:13.972005 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:47:13.972061 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:13.972068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:13.972282 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:13.972421 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:13.973944 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.974973 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.976455 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:13.977140 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:13.977248 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:13.981400 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:13.981415 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:13.981429 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:13.981494 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:13.984446 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:13.985232 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:13.985241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:13.985243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:13.985390 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:13.985797 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:13.985800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:13.985802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:13.986141 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:13.986377 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:13.986654 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 10:47:13.986867 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":348,"threads_started":1}
I20260928 10:47:13.988382 21641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:14.071410 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 21513
I20260928 10:47:14.085734 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 21652
Sep 28 10:47:14 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:47:14.101330 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:14.086675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.8.218.194:33129)
0928 10:47:14.087217 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:14.087224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:14.087248 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:47:14.087305 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:14.087311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:14.087512 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:14.087646 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:14.088918 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:14.089997 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:14.091273 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:14.091800 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:14.091895 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:14.095864 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:14.095876 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:14.095891 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:14.095929 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:14.098836 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:14.099555 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:14.099562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:14.099564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:14.099690 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:14.099979 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:14.099982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:14.099984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:14.100307 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:14.100657 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:14.100782 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 10:47:14.100989 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":275,"threads_started":1}
I20260928 10:47:14.102701 21784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:14.186092 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 21652
I20260928 10:47:14.199249 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 21795
Sep 28 10:47:14 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:47:14.216161 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:14.200279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.8.218.195:42273)
0928 10:47:14.200906 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:14.200913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:14.200944 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:47:14.201003 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:14.201009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:14.201244 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:14.201395 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:14.202809 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:14.203855 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:14.205266 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:14.205904 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:14.206028 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:14.210414 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:14.210428 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:14.210445 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:14.210491 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:14.213173 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:14.214012 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:14.214021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:14.214023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:14.214187 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:14.214577 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:14.214581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:14.214583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:14.214948 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:14.215261 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:14.215548 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 10:47:14.215805 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":335,"threads_started":1}
I20260928 10:47:14.217589 21918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:14.300952 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 21795
I20260928 10:47:14.313108 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 21401
I20260928 10:47:14.314479 21463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:14.418660 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 21401
I20260928 10:47:14.437578 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45579
--webserver_interface=127.8.218.254
--webserver_port=36171
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:45579
--ranger_config_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:40207/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.218.212:40207/kms with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:14.756788 21935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:14.757090 21935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:14.757225 21935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:14.764683 21935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:14.764802 21935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:14.764870 21935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:14.764945 21935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:14.765007 21935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:14.775404 21935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.218.212:40207/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:45579
--ranger_config_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=36171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:14.776800 21935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:14.778235 21935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:14.790736 21941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:14.791446 21940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:14.792229 21943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:14.792973 21935 server_base.cc:1034] running on GCE node
I20260928 10:47:14.794018 21935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:14.795948 21935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:14.797196 21935 hybrid_clock.cc:648] HybridClock initialized: now 1790592434797132 us; error 60 us; skew 500 ppm
I20260928 10:47:14.797662 21935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:14 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:14.800709 21935 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:47:14.802278 21935 webserver.cc:460] Webserver started at http://127.8.218.254:36171/ using document root <none> and password file <none>
I20260928 10:47:14.802953 21935 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:14.803063 21935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 10:47:14 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:14.880776 21935 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 10:47:14.883906 21951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:14.885326 21935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:47:14.885550 21935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "7bf1b9490d25423f966d7812f1064c1f"
format_stamp: "Formatted at 2026-09-28 10:47:10 on dist-test-slave-mn38"
server_key: "4cc51df7f408c3f071ac05df6ede6c3d"
server_key_iv: "2bcfdb54bdeb04b15442de02564f222c"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:14.886539 21935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:14.905812 21935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:14.910003 21935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:14.910290 21935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:14.928501 21935 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:45579
I20260928 10:47:14.928511 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:45579 every 8 connection(s)
I20260928 10:47:14.929989 21935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:47:14.936874 22003 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:47:14.940491 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Bootstrap starting.
I20260928 10:47:14.940567 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 21935
I20260928 10:47:14.941546 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:40207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:46577
--local_ip_for_outbound_sockets=127.8.218.193
--tserver_master_addrs=127.8.218.254:45579
--webserver_port=33257
--webserver_interface=127.8.218.193
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.218.212:40207/kms with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:47:14.962107 22003 log.cc:824] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:14.971184 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:47:14.971696 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Bootstrap complete.
I20260928 10:47:14.980360 22003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } }
I20260928 10:47:14.981216 22003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf1b9490d25423f966d7812f1064c1f, State: Initialized, Role: FOLLOWER
I20260928 10:47:14.981945 22003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } }
I20260928 10:47:14.982162 22003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:47:14.982273 22003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:47:14.982429 22003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:47:14.986442 22003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } }
I20260928 10:47:14.986943 22003 leader_election.cc:302] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf1b9490d25423f966d7812f1064c1f; no voters:
I20260928 10:47:14.987536 22003 leader_election.cc:288] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:47:14.987674 22007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:47:14.988740 22007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [term 2 LEADER]: Becoming Leader. State: Replica: 7bf1b9490d25423f966d7812f1064c1f, State: Running, Role: LEADER
I20260928 10:47:14.989372 22007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } }
I20260928 10:47:14.989749 22003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:14.992043 22009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bf1b9490d25423f966d7812f1064c1f. Latest consensus state: current_term: 2 leader_uuid: "7bf1b9490d25423f966d7812f1064c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } } }
I20260928 10:47:14.992449 22009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:14.993081 22008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7bf1b9490d25423f966d7812f1064c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf1b9490d25423f966d7812f1064c1f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45579 } } }
I20260928 10:47:14.993381 22008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:15.003000 22013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:47:15.009312 22013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:47:15.010947 22013 catalog_manager.cc:1259] Loaded cluster ID: e114ca88d4c3433284db02882aab39f2
I20260928 10:47:15.011178 22013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:47:15.015396 22013 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:47:15.017261 22013 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Loaded TSK: 0
I20260928 10:47:15.019268 22013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:47:15.022552 22003 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 10:47:15.386377 22005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:15.386897 22005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:15.387110 22005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:15.398589 22005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:15.399004 22005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:15.399279 22005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:47:15.413157 22005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.218.212:40207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:46577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=33257
--tserver_master_addrs=127.8.218.254:45579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:15.414758 22005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:15.416252 22005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:15.445181 22041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:15.447254 22040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:15.455124 22044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:15.455390 22005 server_base.cc:1034] running on GCE node
I20260928 10:47:15.456197 22005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:15.457607 22005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:15.458863 22005 hybrid_clock.cc:648] HybridClock initialized: now 1790592435458798 us; error 50 us; skew 500 ppm
I20260928 10:47:15.459358 22005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:15 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:15.462961 22005 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:47:15.464871 22005 webserver.cc:460] Webserver started at http://127.8.218.193:33257/ using document root <none> and password file <none>
I20260928 10:47:15.465713 22005 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:15.465860 22005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 10:47:15 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:15.493300 22005 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.007s
I20260928 10:47:15.504659 22051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:15.516399 22005 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.003s sys 0.003s
I20260928 10:47:15.516728 22005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "f33f2af14f114175bb862665fd6d4cac"
format_stamp: "Formatted at 2026-09-28 10:47:12 on dist-test-slave-mn38"
server_key: "0bcca29411923c97186ddf4cb692a65d"
server_key_iv: "fd83a374a8ee8297f6858304d0043ed8"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:15.521130 22005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:15.581105 22005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:15.591084 22005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:15.591377 22005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:15.592449 22005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:15.594015 22005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:15.594116 22005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:15.594230 22005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:15.594306 22005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:15.640584 22005 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:46577
I20260928 10:47:15.640650 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:46577 every 8 connection(s)
I20260928 10:47:15.642267 22005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:47:15.644340 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 22005
I20260928 10:47:15.645231 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:40207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:33129
--local_ip_for_outbound_sockets=127.8.218.194
--tserver_master_addrs=127.8.218.254:45579
--webserver_port=46413
--webserver_interface=127.8.218.194
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.218.212:40207/kms with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:47:15 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:15.695876 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:15.651791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45579 (local address 127.8.218.193:53241)
0928 10:47:15.655642 (+ 3851us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:15.655697 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:15.658614 (+ 2917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:15.663540 (+ 4926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:15.663552 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:15.664224 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:15.665098 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:15.665131 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:15.667704 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:15.667711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:15.669930 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:15.669965 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:15.672136 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:15.672146 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:15.672473 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:15.678085 (+ 5612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:15.678129 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:15.683621 (+ 5492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:15.689872 (+ 6251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:15.689931 (+ 59us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:15.689952 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:15.690824 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:15.691558 (+ 734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:15.691564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:15.691568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:15.691877 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:15.693039 (+ 1162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:15.693047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:15.693493 (+ 446us) client_negotiation.cc:769] Sending connection context
0928 10:47:15.693852 (+ 359us) client_negotiation.cc:240] Negotiation successful
0928 10:47:15.694448 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3495,"thread_start_us":3257,"threads_started":1}
I20260928 10:47:15.695930 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:15.653252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:53241 (local address 127.8.218.254:45579)
0928 10:47:15.654960 (+ 1708us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:15.654983 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:15.655896 (+ 913us) server_negotiation.cc:407] Connection header received
0928 10:47:15.659523 (+ 3627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:15.659547 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:15.662094 (+ 2547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:15.662710 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:15.665419 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:15.667407 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:15.670199 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:15.671414 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:15.671656 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:15.684312 (+ 12656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:15.684328 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:15.684353 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:15.684420 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:15.689380 (+ 4960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:15.691104 (+ 1724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:15.691115 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:15.691124 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:15.691375 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:15.692042 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:15.692047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:15.692051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:15.692846 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:15.693054 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:15.694154 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 10:47:15.694639 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":1125,"threads_started":1}
I20260928 10:47:15.699824 22165 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45579
I20260928 10:47:15.700220 22165 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:15.701215 22165 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
W20260928 10:47:15.705801 22165 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:45579 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:47:15.706148 22165 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:15.706712 22165 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
I20260928 10:47:15.708774 22165 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:15.709160 22165 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
W20260928 10:47:15.710690 22165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:47:16.048044 22169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:16.048349 22169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:16.048486 22169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:16.056370 22169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:16.056483 22169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:16.056639 22169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:47:16.066870 22169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.218.212:40207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:33129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=46413
--tserver_master_addrs=127.8.218.254:45579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:16.068176 22169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:16.069600 22169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:16.085660 22184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:16.086272 22182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:16.087450 22181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:16.088076 22169 server_base.cc:1034] running on GCE node
I20260928 10:47:16.088779 22169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:16.089845 22169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:16.091169 22169 hybrid_clock.cc:648] HybridClock initialized: now 1790592436091066 us; error 94 us; skew 500 ppm
I20260928 10:47:16.091526 22169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:16 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:16.094750 22169 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:47:16.098718 22169 webserver.cc:460] Webserver started at http://127.8.218.194:46413/ using document root <none> and password file <none>
I20260928 10:47:16.099416 22169 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:16.099555 22169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 10:47:16 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:16.123631 22169 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.006s
I20260928 10:47:16.127682 22194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:16.129616 22169 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:47:16.129822 22169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "1456460f70f74dee9b4524fa998686b5"
format_stamp: "Formatted at 2026-09-28 10:47:13 on dist-test-slave-mn38"
server_key: "84e6ac3aa281c01790eef31d2d1779e0"
server_key_iv: "f6ee01f2c708332654e9734236b59aec"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:16.131338 22169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:16.163529 22169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:16.169219 22169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:16.169690 22169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:16.179281 22169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:16.180977 22169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:16.181097 22169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:16.181218 22169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:16.181293 22169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:16.257031 22169 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:33129
I20260928 10:47:16.257377 22308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:33129 every 8 connection(s)
I20260928 10:47:16.258991 22169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:47:16.267944 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 22169
I20260928 10:47:16.268760 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.218.212:40207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:42273
--local_ip_for_outbound_sockets=127.8.218.195
--tserver_master_addrs=127.8.218.254:45579
--webserver_port=40417
--webserver_interface=127.8.218.195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.218.212:40207/kms with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:47:16 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 10:47:16 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:16.299194 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:16.263757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45579 (local address 127.8.218.194:51071)
0928 10:47:16.269568 (+ 5811us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:16.269754 (+ 186us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:16.272373 (+ 2619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:16.277957 (+ 5584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:16.277971 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:16.278751 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:16.279720 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:16.279758 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:16.281344 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:16.281350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:16.283436 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:16.283458 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:16.284773 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:16.284781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:16.285105 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:16.285970 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:16.286006 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:16.288714 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:16.293124 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:16.293136 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:16.293159 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:16.293931 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:16.294682 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:16.294689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:16.294693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:16.295072 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:16.295973 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:16.295982 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:16.296535 (+ 553us) client_negotiation.cc:769] Sending connection context
0928 10:47:16.296899 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 10:47:16.297617 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":1309,"threads_started":1}
I20260928 10:47:16.300468 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:16.265041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:51071 (local address 127.8.218.254:45579)
0928 10:47:16.276880 (+ 11839us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:16.276886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:16.276922 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:47:16.276995 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:16.277000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:16.277109 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:16.277261 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:16.279982 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:16.281089 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:16.283654 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:16.284301 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:16.284426 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:16.289052 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:16.289060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:16.289064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:16.289111 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:16.292854 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:16.294295 (+ 1441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:16.294302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:16.294306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:16.294504 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:16.295245 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:16.295251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:16.295254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:16.295677 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:16.295861 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:16.299787 (+ 3926us) server_negotiation.cc:299] Negotiation successful
0928 10:47:16.300091 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11686,"thread_start_us":12332,"threads_started":1}
I20260928 10:47:16.302881 22309 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45579
I20260928 10:47:16.303216 22309 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:16.303977 22309 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
I20260928 10:47:16.306035 21968 ts_manager.cc:194] Registered new tserver with Master: 1456460f70f74dee9b4524fa998686b5 (127.8.218.194:33129)
I20260928 10:47:16.308264 21968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:51071
I20260928 10:47:16.710832 22165 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:16.711268 22165 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
I20260928 10:47:16.712627 21968 ts_manager.cc:194] Registered new tserver with Master: f33f2af14f114175bb862665fd6d4cac (127.8.218.193:46577)
I20260928 10:47:16.713980 21968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:53241
W20260928 10:47:16.720172 22313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:16.720587 22313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:16.720788 22313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:16.729272 22313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:16.729513 22313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:16.729775 22313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:47:16.745234 22313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.218.212:40207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:42273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=40417
--tserver_master_addrs=127.8.218.254:45579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:16.747036 22313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:16.748806 22313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 10:47:16 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
W20260928 10:47:16.778081 22319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:16.814894 22320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:16.833259 22313 server_base.cc:1034] running on GCE node
W20260928 10:47:16.836134 22322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:16.837198 22313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:16.841076 22313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:16.842417 22313 hybrid_clock.cc:648] HybridClock initialized: now 1790592436842336 us; error 76 us; skew 500 ppm
I20260928 10:47:16.842872 22313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:16 dist-test-slave-mn38 krb5kdc[20957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:16.847313 22313 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:47:16.849406 22313 webserver.cc:460] Webserver started at http://127.8.218.195:40417/ using document root <none> and password file <none>
I20260928 10:47:16.850425 22313 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:16.850602 22313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 10:47:16 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:16.875610 22313 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:47:16.879547 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:16.881598 22313 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:47:16.881815 22313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "cd5891da25f1459aa63eb0b608f4c2a3"
format_stamp: "Formatted at 2026-09-28 10:47:13 on dist-test-slave-mn38"
server_key: "8a1ed90d85feb1db67862a57d70b7b3c"
server_key_iv: "e54e2e394c3a0b4aa162e8f653f7eeb3"
server_key_version: "kuduclusterkey@0"
I20260928 10:47:16.883306 22313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:16.903426 22313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:16.907661 22313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:16.907931 22313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:16.908982 22313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:16.910581 22313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:16.910681 22313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:16.910787 22313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:16.910854 22313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:16.978680 22313 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:42273
I20260928 10:47:16.978720 22442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:42273 every 8 connection(s)
I20260928 10:47:16.980257 22313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:47:16.988226 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 22313
Sep 28 10:47:16 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:17.010180 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:16.984452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:52993 (local address 127.8.218.254:45579)
0928 10:47:16.985851 (+ 1399us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:16.985858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:16.986988 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 10:47:16.989242 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:16.989248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:16.989344 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:16.989479 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:16.991913 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:16.993193 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:16.996177 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:16.996871 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:16.996994 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.002214 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:17.002221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:17.002225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:17.002264 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:17.005803 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.006890 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.006896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.006899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.007085 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.007689 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.007695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.007698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.008080 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:17.008214 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.009251 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.009521 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":1148,"threads_started":1}
I20260928 10:47:17.012442 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:16.983936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45579 (local address 127.8.218.195:52993)
0928 10:47:16.986411 (+ 2475us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:16.986471 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:16.988756 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:16.990012 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:16.990028 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:16.990737 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:16.991659 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:16.991688 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:16.993469 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:16.993475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:16.995930 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:16.995953 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:16.997465 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:16.997473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:16.997790 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:16.998658 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:16.998689 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:17.001883 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:17.006050 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:17.006061 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:17.006083 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:17.006725 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:17.007247 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:17.007252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:17.007256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:17.007557 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:17.008305 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:17.008313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:17.008789 (+ 476us) client_negotiation.cc:769] Sending connection context
0928 10:47:17.010414 (+ 1625us) client_negotiation.cc:240] Negotiation successful
0928 10:47:17.011037 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165,"thread_start_us":1942,"threads_started":1}
I20260928 10:47:17.014410 22443 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45579
I20260928 10:47:17.014730 22443 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:17.015647 22443 heartbeater.cc:511] Master 127.8.218.254:45579 requested a full tablet report, sending...
I20260928 10:47:17.018112 21968 ts_manager.cc:194] Registered new tserver with Master: cd5891da25f1459aa63eb0b608f4c2a3 (127.8.218.195:42273)
I20260928 10:47:17.019253 21968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:52993
I20260928 10:47:17.022810 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:16.989234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.8.218.254:45579)
0928 10:47:17.000801 (+ 11567us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.000807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.001715 (+ 908us) server_negotiation.cc:407] Connection header received
0928 10:47:17.001799 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.001804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.001890 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.001988 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:17.004414 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.005774 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.007862 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.008552 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.008678 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.012564 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:17.012572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:17.012577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:17.012621 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:17.018949 (+ 6328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.019908 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.019914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.019917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.020066 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.020497 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.020501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.020504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.020835 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:17.021085 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.022309 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.022530 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11400,"thread_start_us":12348,"threads_started":1}
I20260928 10:47:17.025312 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:47:17.049978 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.028299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.8.218.254:45579)
0928 10:47:17.028470 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.028477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.034019 (+ 5542us) server_negotiation.cc:407] Connection header received
0928 10:47:17.034197 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.034202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.034285 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.034370 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:17.035798 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.036614 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.038312 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.038739 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.038849 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.042137 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:17.042144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:17.042148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:17.042194 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:17.045854 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.046855 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.046862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.046866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.047038 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.047489 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.047495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.047499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.047791 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:17.048018 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.049318 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.049642 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:47:17.055841 21968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:47:17.066694 21968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 28 10:47:17 dist-test-slave-mn38 krb5kdc[20957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for HTTP/127.8.218.212@KRBTEST.COM
I20260928 10:47:17.178918 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.092342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:46577 (local address 127.0.0.1:59742)
0928 10:47:17.112729 (+ 20387us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:17.112783 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:17.113012 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:17.170220 (+ 57208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:17.170228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:17.170272 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:17.170783 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.170802 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.173163 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.173170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.175681 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.175693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.177907 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.177915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.178138 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.178212 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 10:47:17.178327 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 10:47:17.178522 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20024,"thread_start_us":19850,"threads_started":1}
I20260928 10:47:17.180913 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.098145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.8.218.193:46577)
0928 10:47:17.169384 (+ 71239us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.169396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.169431 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:47:17.169586 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.169596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.169928 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.170177 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:17.171028 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.172660 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.176081 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.177647 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.180037 (+ 2390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.180297 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.180372 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.180549 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71022,"thread_start_us":72633,"threads_started":1}
I20260928 10:47:17.186245 22100 tablet_service.cc:1467] Processing CreateTablet for tablet ab38b7134bba42259c00e46640c078c7 (DEFAULT_TABLE table=test-table [id=92283bbc7e844cf5b4060c33bfd85656]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:17.187949 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab38b7134bba42259c00e46640c078c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:17.190277 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.093198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33129 (local address 127.0.0.1:38230)
0928 10:47:17.179213 (+ 86015us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:17.179234 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:17.179357 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:17.183173 (+ 3816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:17.183178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:17.183208 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:17.183385 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.183396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.185603 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.185609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.187367 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.187384 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.189391 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.189399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.189632 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.189671 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:47:17.189750 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 10:47:17.189928 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85892,"thread_start_us":88234,"threads_started":1}
I20260928 10:47:17.191594 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.098553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.8.218.194:33129)
0928 10:47:17.170770 (+ 72217us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.170779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.181389 (+ 10610us) server_negotiation.cc:407] Connection header received
0928 10:47:17.181551 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.181561 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.181849 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.183734 (+ 1885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:17.183982 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.185411 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.187604 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.189069 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.190806 (+ 1737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.191022 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.191109 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.191279 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72042,"thread_start_us":72445,"threads_started":1}
I20260928 10:47:17.196383 22244 tablet_service.cc:1467] Processing CreateTablet for tablet ab38b7134bba42259c00e46640c078c7 (DEFAULT_TABLE table=test-table [id=92283bbc7e844cf5b4060c33bfd85656]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:17.198324 22244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab38b7134bba42259c00e46640c078c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:17.206875 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.092618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:42273 (local address 127.0.0.1:39306)
0928 10:47:17.168570 (+ 75952us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:17.168596 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:17.168764 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:17.175005 (+ 6241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:17.175010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:17.175042 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:17.175427 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.175439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.197794 (+ 22355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.197801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.200109 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.200130 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.204555 (+ 4425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.204562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.206237 (+ 1675us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.206275 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:47:17.206364 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 10:47:17.206515 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75757,"thread_start_us":99405,"threads_started":1}
I20260928 10:47:17.208019 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.092871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.8.218.195:42273)
0928 10:47:17.174049 (+ 81178us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.174058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.174105 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:47:17.174206 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.174215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.174505 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.174718 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:17.182410 (+ 7692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.194483 (+ 12073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.202354 (+ 7871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.203950 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.204081 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.204310 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.207282 (+ 2972us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.207675 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80988,"thread_start_us":82868,"threads_started":1}
I20260928 10:47:17.213187 22378 tablet_service.cc:1467] Processing CreateTablet for tablet ab38b7134bba42259c00e46640c078c7 (DEFAULT_TABLE table=test-table [id=92283bbc7e844cf5b4060c33bfd85656]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:17.215286 22378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab38b7134bba42259c00e46640c078c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:17.244266 22466 tablet_bootstrap.cc:492] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Bootstrap starting.
I20260928 10:47:17.249061 22467 tablet_bootstrap.cc:492] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Bootstrap starting.
I20260928 10:47:17.250326 22466 tablet_bootstrap.cc:654] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:17.251786 22466 log.cc:824] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:17.252506 22467 tablet_bootstrap.cc:654] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:17.256394 22467 log.cc:824] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:17.260066 22466 tablet_bootstrap.cc:492] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: No bootstrap required, opened a new log
I20260928 10:47:17.260418 22466 ts_tablet_manager.cc:1397] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260928 10:47:17.271466 22467 tablet_bootstrap.cc:492] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: No bootstrap required, opened a new log
I20260928 10:47:17.272025 22467 ts_tablet_manager.cc:1397] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.002s
I20260928 10:47:17.274945 22466 raft_consensus.cc:348] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.275595 22466 raft_consensus.cc:374] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:17.275736 22466 raft_consensus.cc:729] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33f2af14f114175bb862665fd6d4cac, State: Initialized, Role: FOLLOWER
I20260928 10:47:17.276648 22466 consensus_queue.cc:260] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.286866 22466 ts_tablet_manager.cc:1428] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Time spent starting tablet: real 0.026s user 0.009s sys 0.012s
I20260928 10:47:17.287366 22165 heartbeater.cc:503] Master 127.8.218.254:45579 was elected leader, sending a full tablet report...
I20260928 10:47:17.289737 22467 raft_consensus.cc:348] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.290395 22467 raft_consensus.cc:374] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:17.290549 22467 raft_consensus.cc:729] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5891da25f1459aa63eb0b608f4c2a3, State: Initialized, Role: FOLLOWER
I20260928 10:47:17.292979 22469 tablet_bootstrap.cc:492] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: Bootstrap starting.
I20260928 10:47:17.293833 22467 consensus_queue.cc:260] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.297840 22443 heartbeater.cc:503] Master 127.8.218.254:45579 was elected leader, sending a full tablet report...
I20260928 10:47:17.298051 22469 tablet_bootstrap.cc:654] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:17.298887 22467 ts_tablet_manager.cc:1428] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Time spent starting tablet: real 0.027s user 0.012s sys 0.001s
I20260928 10:47:17.299443 22469 log.cc:824] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:17.306715 22469 tablet_bootstrap.cc:492] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: No bootstrap required, opened a new log
I20260928 10:47:17.307139 22469 ts_tablet_manager.cc:1397] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260928 10:47:17.313151 22309 heartbeater.cc:503] Master 127.8.218.254:45579 was elected leader, sending a full tablet report...
I20260928 10:47:17.318881 22469 raft_consensus.cc:348] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.319428 22469 raft_consensus.cc:374] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:17.319617 22469 raft_consensus.cc:729] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1456460f70f74dee9b4524fa998686b5, State: Initialized, Role: FOLLOWER
I20260928 10:47:17.320469 22469 consensus_queue.cc:260] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.328416 22469 ts_tablet_manager.cc:1428] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: Time spent starting tablet: real 0.021s user 0.013s sys 0.007s
W20260928 10:47:17.401243 22166 tablet.cc:2369] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:17.412001 22472 raft_consensus.cc:482] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:47:17.412366 22472 raft_consensus.cc:504] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.414624 22472 leader_election.cc:288] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f33f2af14f114175bb862665fd6d4cac (127.8.218.193:46577), 1456460f70f74dee9b4524fa998686b5 (127.8.218.194:33129)
I20260928 10:47:17.422930 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.414973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:46577 (local address 127.8.218.195:41555)
0928 10:47:17.415435 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:17.415455 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:17.415579 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:17.416444 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:17.416450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:17.416480 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:17.416684 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.416698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.418211 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.418217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.420290 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.420308 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.422117 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.422123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.422327 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.422363 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:47:17.422468 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 10:47:17.422627 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321}
I20260928 10:47:17.423972 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.415760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:41555 (local address 127.8.218.193:46577)
0928 10:47:17.415964 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.415969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.415990 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:47:17.416064 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.416069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.416157 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.416256 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:17.416863 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.418011 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.420497 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.421759 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.421923 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.422004 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.423567 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.423724 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:47:17.425284 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab38b7134bba42259c00e46640c078c7" candidate_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33f2af14f114175bb862665fd6d4cac" is_pre_election: true
I20260928 10:47:17.426003 22120 raft_consensus.cc:2454] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5891da25f1459aa63eb0b608f4c2a3 in term 0.
I20260928 10:47:17.427239 22331 leader_election.cc:302] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd5891da25f1459aa63eb0b608f4c2a3, f33f2af14f114175bb862665fd6d4cac; no voters:
I20260928 10:47:17.428012 22472 raft_consensus.cc:2781] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:47:17.428180 22472 raft_consensus.cc:482] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:47:17.428270 22472 raft_consensus.cc:3037] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:17.430796 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.416466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:37151 (local address 127.8.218.194:33129)
0928 10:47:17.416646 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.416651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.423507 (+ 6856us) server_negotiation.cc:407] Connection header received
0928 10:47:17.423603 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.423608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.423708 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.423803 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:17.424399 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.425545 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.427965 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.429283 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.429567 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.429661 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.430352 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.430520 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:47:17.431564 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.416054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:33129 (local address 127.8.218.195:37151)
0928 10:47:17.423240 (+ 7186us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:17.423261 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:17.423383 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:17.423989 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:17.423994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:17.424025 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:17.424212 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.424224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.425753 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.425759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.427747 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:17.427767 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.429865 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.429872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:17.430135 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.430174 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:47:17.431071 (+ 897us) client_negotiation.cc:240] Negotiation successful
0928 10:47:17.431226 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7064,"threads_started":1}
I20260928 10:47:17.431523 22472 raft_consensus.cc:504] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.433189 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab38b7134bba42259c00e46640c078c7" candidate_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33f2af14f114175bb862665fd6d4cac"
I20260928 10:47:17.433723 22120 raft_consensus.cc:3037] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:17.437222 22472 leader_election.cc:288] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [CANDIDATE]: Term 1 election: Requested vote from peers f33f2af14f114175bb862665fd6d4cac (127.8.218.193:46577), 1456460f70f74dee9b4524fa998686b5 (127.8.218.194:33129)
I20260928 10:47:17.437402 22264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab38b7134bba42259c00e46640c078c7" candidate_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1456460f70f74dee9b4524fa998686b5" is_pre_election: true
I20260928 10:47:17.438266 22264 raft_consensus.cc:2454] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5891da25f1459aa63eb0b608f4c2a3 in term 0.
I20260928 10:47:17.438949 22263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab38b7134bba42259c00e46640c078c7" candidate_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1456460f70f74dee9b4524fa998686b5"
I20260928 10:47:17.439222 22263 raft_consensus.cc:3037] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:17.442461 22263 raft_consensus.cc:2454] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5891da25f1459aa63eb0b608f4c2a3 in term 1.
I20260928 10:47:17.443117 22333 leader_election.cc:302] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1456460f70f74dee9b4524fa998686b5, cd5891da25f1459aa63eb0b608f4c2a3; no voters:
I20260928 10:47:17.443183 22120 raft_consensus.cc:2454] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5891da25f1459aa63eb0b608f4c2a3 in term 1.
I20260928 10:47:17.443629 22472 raft_consensus.cc:2781] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:17.445011 22472 raft_consensus.cc:686] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 1 LEADER]: Becoming Leader. State: Replica: cd5891da25f1459aa63eb0b608f4c2a3, State: Running, Role: LEADER
I20260928 10:47:17.445673 22472 consensus_queue.cc:237] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } }
I20260928 10:47:17.450778 21967 catalog_manager.cc:5495] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd5891da25f1459aa63eb0b608f4c2a3 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5891da25f1459aa63eb0b608f4c2a3" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 } health_report { overall_health: UNKNOWN } } }
I20260928 10:47:17.469851 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.463651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.8.218.195:42273)
0928 10:47:17.463878 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.463885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.463909 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:47:17.464014 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.464019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.464133 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.464303 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:47:17.464723 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.465894 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.467249 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.467871 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.468000 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.468485 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:47:17.469236 (+ 751us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:47:17.469406 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.469494 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.469581 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 10:47:17.479439 22263 raft_consensus.cc:1261] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 1 FOLLOWER]: Refusing update from remote peer cd5891da25f1459aa63eb0b608f4c2a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:17.479439 22120 raft_consensus.cc:1261] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 1 FOLLOWER]: Refusing update from remote peer cd5891da25f1459aa63eb0b608f4c2a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:17.480702 22478 consensus_queue.cc:1035] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1456460f70f74dee9b4524fa998686b5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:17.481166 22472 consensus_queue.cc:1035] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f33f2af14f114175bb862665fd6d4cac" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 46577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 10:47:17.483531 22444 tablet.cc:2369] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:17.517407 22488 mvcc.cc:205] Tried to move back new op lower bound from 7334266623904366592 to 7334266623781982208. Current Snapshot: MvccSnapshot[applied={T|T < 7334266623904366592 or (T in {7334266623904366592})}]
I20260928 10:47:17.525913 21967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:47:17.526274 21967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:47:17.530944 21967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bf1b9490d25423f966d7812f1064c1f: Sending DeleteTablet for 3 replicas of tablet ab38b7134bba42259c00e46640c078c7
I20260928 10:47:17.532248 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet ab38b7134bba42259c00e46640c078c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:17 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:59742
I20260928 10:47:17.533741 22378 tablet_service.cc:1514] Processing DeleteTablet for tablet ab38b7134bba42259c00e46640c078c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:17 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:39306
I20260928 10:47:17.533813 22244 tablet_service.cc:1514] Processing DeleteTablet for tablet ab38b7134bba42259c00e46640c078c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:17 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:38230
I20260928 10:47:17.536047 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 22005
I20260928 10:47:17.548322 22496 tablet_replica.cc:330] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: stopping tablet replica
I20260928 10:47:17.548880 22496 raft_consensus.cc:2229] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:47:17.553325 22497 tablet_replica.cc:330] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: stopping tablet replica
I20260928 10:47:17.553912 22497 raft_consensus.cc:2229] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:47:17.554569 22497 raft_consensus.cc:2258] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:17.555121 22496 raft_consensus.cc:2258] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:17.557571 22496 ts_tablet_manager.cc:1905] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:17.560021 22495 tablet_replica.cc:330] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: stopping tablet replica
I20260928 10:47:17.560700 22495 raft_consensus.cc:2229] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:47:17.561403 22495 raft_consensus.cc:2258] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:17.563802 22495 ts_tablet_manager.cc:1905] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:17.565738 22497 ts_tablet_manager.cc:1905] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:17.571213 22495 ts_tablet_manager.cc:1918] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:17.571508 22495 log.cc:1196] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/ab38b7134bba42259c00e46640c078c7
I20260928 10:47:17.572139 22495 ts_tablet_manager.cc:1939] T ab38b7134bba42259c00e46640c078c7 P f33f2af14f114175bb862665fd6d4cac: Deleting consensus metadata
I20260928 10:47:17.574455 21952 catalog_manager.cc:4841] TS f33f2af14f114175bb862665fd6d4cac (127.8.218.193:46577): tablet ab38b7134bba42259c00e46640c078c7 (table test-table [id=92283bbc7e844cf5b4060c33bfd85656]) successfully deleted
I20260928 10:47:17.577597 22496 ts_tablet_manager.cc:1918] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:17.577903 22496 log.cc:1196] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/ab38b7134bba42259c00e46640c078c7
I20260928 10:47:17.578318 22497 ts_tablet_manager.cc:1918] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:17.578493 22497 log.cc:1196] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/ab38b7134bba42259c00e46640c078c7
I20260928 10:47:17.578657 22496 ts_tablet_manager.cc:1939] T ab38b7134bba42259c00e46640c078c7 P cd5891da25f1459aa63eb0b608f4c2a3: Deleting consensus metadata
I20260928 10:47:17.579046 22497 ts_tablet_manager.cc:1939] T ab38b7134bba42259c00e46640c078c7 P 1456460f70f74dee9b4524fa998686b5: Deleting consensus metadata
I20260928 10:47:17.580734 21955 catalog_manager.cc:4841] TS cd5891da25f1459aa63eb0b608f4c2a3 (127.8.218.195:42273): tablet ab38b7134bba42259c00e46640c078c7 (table test-table [id=92283bbc7e844cf5b4060c33bfd85656]) successfully deleted
I20260928 10:47:17.580884 21954 catalog_manager.cc:4841] TS 1456460f70f74dee9b4524fa998686b5 (127.8.218.194:33129): tablet ab38b7134bba42259c00e46640c078c7 (table test-table [id=92283bbc7e844cf5b4060c33bfd85656]) successfully deleted
I20260928 10:47:17.584739 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.536984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.8.218.193:46577)
0928 10:47:17.537179 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.537185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.550024 (+ 12839us) server_negotiation.cc:407] Connection header received
0928 10:47:17.550197 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.550204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.550305 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.550430 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:17.552673 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.553979 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.563931 (+ 9952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.564586 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.564737 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.573103 (+ 8366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:17.573117 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:17.573135 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:17.573186 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:17.576895 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.582085 (+ 5190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.582098 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.582102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.582319 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.582907 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.582920 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.582924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.583454 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:17.583762 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.584109 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.584359 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:47:17.585705 22160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:17.749819 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 22005
I20260928 10:47:17.770754 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 22169
I20260928 10:47:17.788343 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.771590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.8.218.194:33129)
0928 10:47:17.771799 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.771806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.771911 (+ 105us) server_negotiation.cc:407] Connection header received
0928 10:47:17.772085 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.772090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.772182 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.772283 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:17.774053 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.775129 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.777041 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.777712 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.777840 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.781180 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:17.781200 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:17.781220 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:17.781267 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:17.785117 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.786052 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.786063 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.786066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.786236 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.786661 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.786665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.786669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.787135 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:17.787414 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.787723 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.787975 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:47:17.789327 22304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:17.918051 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 22169
I20260928 10:47:17.934275 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 22313
I20260928 10:47:17.952564 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:17.935052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.8.218.195:42273)
0928 10:47:17.935237 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:17.935243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:17.935318 (+ 75us) server_negotiation.cc:407] Connection header received
0928 10:47:17.935506 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:17.935512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:17.935614 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:17.935723 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:17.937380 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.938598 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.940521 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:17.941224 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:17.941376 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:17.944777 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:17.944792 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:17.944812 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:17.944864 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:17.949042 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.950142 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.950153 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.950157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.950350 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:17.950777 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:17.950782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:17.950785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:17.951286 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:17.951740 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:17.951903 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 10:47:17.952138 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:47:17.953406 22438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:18.077615 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 22313
I20260928 10:47:18.093531 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 21935
I20260928 10:47:18.094702 21998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:18.206180 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 21935
I20260928 10:47:18.228621 9067 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 10:47:18.564627 9067 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 10:47:18.564823 9067 mini_ranger.cc:66] Stopping Ranger...
I20260928 10:47:18.900837 9067 mini_ranger.cc:68] Stopped Ranger
2026-09-28 10:47:18.901 UTC [20987] LOG: received smart shutdown request
2026-09-28 10:47:18.904 UTC [20987] LOG: background worker "logical replication launcher" (PID 20996) exited with exit code 1
2026-09-28 10:47:18.904 UTC [20991] LOG: shutting down
2026-09-28 10:47:19.101 UTC [20987] LOG: database system is shut down
2026-09-28T10:47:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54198 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:47:19 dist-test-slave-mn38 krb5kdc[22511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:47:19 dist-test-slave-mn38 krb5kdc[22511](info): set up 2 sockets
Sep 28 10:47:19 dist-test-slave-mn38 krb5kdc[22511](info): commencing operation
krb5kdc: starting...
W20260928 10:47:20.239089 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:47:20 dist-test-slave-mn38 krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:47:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:47:20.473732 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36007
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:36007,127.8.218.253:43637,127.8.218.252:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:20.820456 22524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:20.820749 22524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:20.820837 22524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:20.828302 22524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:20.828436 22524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:20.828491 22524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:20.828547 22524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:20.828588 22524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:20.838793 22524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:36007,127.8.218.253:43637,127.8.218.252:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:20.840098 22524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:20.841357 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:20.853168 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:20.853168 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:20.853205 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:20.853820 22524 server_base.cc:1034] running on GCE node
I20260928 10:47:20.854671 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:20.856257 22524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:20.857528 22524 hybrid_clock.cc:648] HybridClock initialized: now 1790592440857474 us; error 45 us; skew 500 ppm
I20260928 10:47:20.857906 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:20 dist-test-slave-mn38 krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592440, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:20.865625 22524 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:47:20.867431 22524 webserver.cc:460] Webserver started at http://127.8.218.254:44137/ using document root <none> and password file <none>
I20260928 10:47:20.868207 22524 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:20.868312 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:20.868578 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:20.871769 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "56e41aad0a2045459eff32202844a210"
format_stamp: "Formatted at 2026-09-28 10:47:20 on dist-test-slave-mn38"
server_key: "07cc03c787ed9b60dd0b1fd1b76ccb92"
server_key_iv: "1a9f74e2129f697908bde799b217dce5"
server_key_version: "encryptionkey@0"
I20260928 10:47:20.872632 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "56e41aad0a2045459eff32202844a210"
format_stamp: "Formatted at 2026-09-28 10:47:20 on dist-test-slave-mn38"
server_key: "07cc03c787ed9b60dd0b1fd1b76ccb92"
server_key_iv: "1a9f74e2129f697908bde799b217dce5"
server_key_version: "encryptionkey@0"
I20260928 10:47:20.878539 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:47:20.882355 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:20.883824 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 10:47:20.883991 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "56e41aad0a2045459eff32202844a210"
format_stamp: "Formatted at 2026-09-28 10:47:20 on dist-test-slave-mn38"
server_key: "07cc03c787ed9b60dd0b1fd1b76ccb92"
server_key_iv: "1a9f74e2129f697908bde799b217dce5"
server_key_version: "encryptionkey@0"
I20260928 10:47:20.884197 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:20.914173 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:20.918722 22524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:20.919013 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:20.937955 22524 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:36007
I20260928 10:47:20.937989 22590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:36007 every 8 connection(s)
I20260928 10:47:20.939429 22524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:47:20.943300 22591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:20.946986 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 22524
I20260928 10:47:20.947180 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:47:20.947855 9067 external_mini_cluster.cc:1428] Setting key 2de629edadc7b14af72135fb9d46e1b8
I20260928 10:47:20.950201 22591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:47:20 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592440, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.253@KRBTEST.COM" created.
I20260928 10:47:20.974383 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:20.954695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.8.218.254:36007)
0928 10:47:20.955363 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:20.955370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:20.955423 (+ 53us) server_negotiation.cc:407] Connection header received
0928 10:47:20.956445 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:20.956452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:20.956755 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:20.956984 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:20.958232 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:20.960321 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:20.962060 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:20.962966 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:20.963168 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:20.966646 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:20.966658 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:20.966677 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:20.966718 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:20.969728 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:20.970622 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:20.970630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:20.970632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:20.970780 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:20.971323 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:20.971328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:20.971330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:20.971980 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:20.972162 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:20.972861 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 10:47:20.973151 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":308,"threads_started":1}
I20260928 10:47:20.974455 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:20.953305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36007 (local address 127.0.0.1:41498)
0928 10:47:20.954128 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:20.954175 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:20.956067 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:20.957009 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:20.957014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:20.957654 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:20.958013 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:20.958035 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:20.960597 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:20.960602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:20.961892 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:20.961908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:20.963213 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:20.963218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:20.963375 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:20.964064 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:20.964099 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:20.966364 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:20.969875 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:20.969886 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:20.969896 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:20.970461 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:20.970895 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:20.970900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:20.970903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:20.971143 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:20.972129 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:20.972137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:20.972394 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 10:47:20.972660 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 10:47:20.973366 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":153,"thread_start_us":335,"threads_started":1}
I20260928 10:47:20.978173 22591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:47:20.979408 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:20.978996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37116)
0928 10:47:20.979170 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:20.979285 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 10:47:20.979733 22543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
I20260928 10:47:20.981366 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:20.981023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37130)
0928 10:47:20.981153 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:20.981232 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 10:47:20.981607 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:43637: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab.
Entry for principal kudu/127.8.218.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab.
I20260928 10:47:21.000291 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.253:43637
--webserver_interface=127.8.218.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab
--principal=kudu/127.8.218.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:36007,127.8.218.253:43637,127.8.218.252:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:47:21.031020 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } attempt: 1
I20260928 10:47:21.032287 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.031791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37136)
0928 10:47:21.032003 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.032125 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 10:47:21.033519 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.033023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37146)
0928 10:47:21.033212 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.033303 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 10:47:21.033748 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:43637: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
I20260928 10:47:21.102020 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } attempt: 2
I20260928 10:47:21.103369 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.102739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37148)
0928 10:47:21.103030 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.103154 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 10:47:21.104959 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.104412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37162)
0928 10:47:21.104601 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.104723 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 10:47:21.105228 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:43637: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
I20260928 10:47:21.196549 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } attempt: 3
I20260928 10:47:21.198086 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.197392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37166)
0928 10:47:21.197733 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.197878 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260928 10:47:21.199339 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.198852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37172)
0928 10:47:21.199036 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.199140 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 10:47:21.199548 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:43637: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
W20260928 10:47:21.337330 22598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:21.337867 22598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:21.338126 22598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 10:47:21.342810 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } attempt: 4
I20260928 10:47:21.344161 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.343513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37180)
0928 10:47:21.343877 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.343995 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260928 10:47:21.345477 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.344918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37188)
0928 10:47:21.345138 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.345267 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 10:47:21.345758 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:43637: Network error: Client connection negotiation failed: client connection to 127.8.218.253:43637: connect: Connection refused (error 111)
W20260928 10:47:21.347080 22598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:21.347185 22598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:21.347237 22598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:21.347294 22598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:21.347342 22598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:21.358220 22598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:36007,127.8.218.253:43637,127.8.218.252:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab
--principal=kudu/127.8.218.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.253:43637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.218.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:21.359582 22598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:21.360994 22598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:21.373844 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:21.373813 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:21.374143 22603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:21.374809 22598 server_base.cc:1034] running on GCE node
I20260928 10:47:21.375669 22598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:21.377372 22598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:21.378696 22598 hybrid_clock.cc:648] HybridClock initialized: now 1790592441378658 us; error 73 us; skew 500 ppm
I20260928 10:47:21.379108 22598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:21 dist-test-slave-mn38 krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592441, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:21.382896 22598 init.cc:377] Logged in from keytab as kudu/127.8.218.253@KRBTEST.COM (short username kudu)
I20260928 10:47:21.384974 22598 webserver.cc:460] Webserver started at http://127.8.218.253:41509/ using document root <none> and password file <none>
I20260928 10:47:21.385901 22598 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:21.386049 22598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:21.386372 22598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:21.390092 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/data/instance:
uuid: "e5826f50106545689215c75454135e60"
format_stamp: "Formatted at 2026-09-28 10:47:21 on dist-test-slave-mn38"
server_key: "329b2905fd20a105cf4d7c8071784ea3"
server_key_iv: "3892a643df303121fbe004df56ffb4b6"
server_key_version: "encryptionkey@0"
I20260928 10:47:21.391104 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/wal/instance:
uuid: "e5826f50106545689215c75454135e60"
format_stamp: "Formatted at 2026-09-28 10:47:21 on dist-test-slave-mn38"
server_key: "329b2905fd20a105cf4d7c8071784ea3"
server_key_iv: "3892a643df303121fbe004df56ffb4b6"
server_key_version: "encryptionkey@0"
I20260928 10:47:21.397730 22598 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:47:21.402020 22613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:21.403810 22598 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:47:21.404028 22598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/wal
uuid: "e5826f50106545689215c75454135e60"
format_stamp: "Formatted at 2026-09-28 10:47:21 on dist-test-slave-mn38"
server_key: "329b2905fd20a105cf4d7c8071784ea3"
server_key_iv: "3892a643df303121fbe004df56ffb4b6"
server_key_version: "encryptionkey@0"
I20260928 10:47:21.404246 22598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:21.428959 22598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:21.434114 22598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:21.434417 22598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:21.455161 22598 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.253:43637
I20260928 10:47:21.455214 22664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.253:43637 every 8 connection(s)
I20260928 10:47:21.456867 22598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
I20260928 10:47:21.462370 22665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:21.462450 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 22598
I20260928 10:47:21.462654 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-1/wal/instance
I20260928 10:47:21.463073 9067 external_mini_cluster.cc:1428] Setting key 18b1032fd70a8b2fe56756aa5b526489
I20260928 10:47:21.471114 22665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.218.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.252@KRBTEST.COM" created.
Sep 28 10:47:21 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592441, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:21.494459 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.474192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.8.218.254:36007)
0928 10:47:21.474430 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:21.474436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:21.476101 (+ 1665us) server_negotiation.cc:407] Connection header received
0928 10:47:21.478264 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:21.478272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:21.478376 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:21.478511 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:21.480254 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.481324 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.483463 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.483911 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.484064 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:21.488049 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:21.488055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:21.488059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:21.488096 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:21.490984 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:21.491946 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:21.491951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:21.491955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:21.492105 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:21.492538 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:21.492542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:21.492545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:21.492897 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:21.493041 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:21.493867 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 10:47:21.494132 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 10:47:21.495357 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.474805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36007 (local address 127.0.0.1:41502)
0928 10:47:21.475890 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.475949 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:21.477923 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:21.478722 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:21.478733 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:21.479359 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:21.480012 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:21.480034 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.481584 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.481589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:21.483243 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:21.483262 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.484191 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.484196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:21.484456 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:21.485166 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:21.485198 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:21.487758 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:21.491165 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:21.491177 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:21.491196 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:21.491779 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:21.492225 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:21.492228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:21.492230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:21.492427 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:21.493043 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:21.493049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:21.493412 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 10:47:21.493736 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 10:47:21.494146 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":385,"threads_started":1}
I20260928 10:47:21.497942 22665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:47:21 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592441, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab.
Entry for principal kudu/127.8.218.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab.
I20260928 10:47:21.514806 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.252:40681
--webserver_interface=127.8.218.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab
--principal=kudu/127.8.218.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:36007,127.8.218.253:43637,127.8.218.252:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:47:21.515417 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.499024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37198)
0928 10:47:21.499330 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.499352 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:21.499506 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:21.500822 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:21.500827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:21.501261 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:21.501420 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:21.501429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.503528 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.503535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:21.504438 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:21.504452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.505589 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.505594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:21.505765 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:21.506412 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:21.506432 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:21.508584 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:21.512402 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:21.512407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:21.512409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:21.512884 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:21.513287 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:21.513290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:21.513292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:21.513427 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:21.514398 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:21.514405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:21.514579 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 10:47:21.514730 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 10:47:21.515043 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260928 10:47:21.515532 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.499412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.8.218.253:43637)
0928 10:47:21.500205 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:21.500215 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:21.500250 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:47:21.500316 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:21.500325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:21.500616 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:21.500798 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:21.501646 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.503325 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.504597 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.505303 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.505456 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:21.508855 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:21.508872 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:21.508889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:21.508933 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:21.512231 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:21.513025 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:21.513032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:21.513035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:21.513164 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:21.513664 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:21.513669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:21.513672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:21.514124 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:21.514383 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:21.514961 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 10:47:21.515169 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":430,"threads_started":1}
I20260928 10:47:21.518231 22665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:47:21.519539 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.518941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52892)
0928 10:47:21.519160 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.519335 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 10:47:21.519827 22617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
I20260928 10:47:21.521608 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.521133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52904)
0928 10:47:21.521313 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.521423 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 10:47:21.521855 22665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
I20260928 10:47:21.571193 22665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 1
I20260928 10:47:21.572574 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.572053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52916)
0928 10:47:21.572291 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.572401 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 10:47:21.573671 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.573256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52928)
0928 10:47:21.573373 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.573497 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 10:47:21.573881 22665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
I20260928 10:47:21.642130 22665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 2
I20260928 10:47:21.643460 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.642917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52936)
0928 10:47:21.643181 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.643273 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 10:47:21.644467 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.644123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52940)
0928 10:47:21.644275 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.644349 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 10:47:21.644670 22665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
I20260928 10:47:21.644982 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } attempt: 5
Sep 28 10:47:21 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592440, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
I20260928 10:47:21.664621 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.645593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37210)
0928 10:47:21.645790 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.645806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:21.645910 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:21.646295 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:21.646299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:21.646678 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:21.646889 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:21.646898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.648304 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.648309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:21.649404 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:21.649423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.650434 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.650439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:21.650630 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:21.655670 (+ 5040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:21.655700 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:21.657924 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:21.661875 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:21.661880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:21.661883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:21.662363 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:21.662813 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:21.662816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:21.662818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:21.662956 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:21.663827 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:21.663830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:21.663954 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 10:47:21.664099 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 10:47:21.664320 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 10:47:21.664803 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.645659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.8.218.253:43637)
0928 10:47:21.645865 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:21.645871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:21.645893 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:47:21.646008 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:21.646013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:21.646109 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:21.646215 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:21.647044 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.648134 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.649632 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:21.650267 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:21.650440 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:21.658226 (+ 7786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:21.658234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:21.658239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:21.658289 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:21.661673 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:21.662533 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:21.662540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:21.662543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:21.662699 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:21.663211 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:21.663216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:21.663219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:21.663719 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:21.663887 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:21.664252 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 10:47:21.664495 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 10:47:21.665840 22591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:47:21.666879 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.666503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52946)
0928 10:47:21.666670 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.666749 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 10:47:21.667821 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.667477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52948)
0928 10:47:21.667650 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.667726 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 10:47:21.668015 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
I20260928 10:47:21.719264 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 1
I20260928 10:47:21.720593 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.720052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52962)
0928 10:47:21.720253 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.720404 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 10:47:21.721715 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.721286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52978)
0928 10:47:21.721526 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.721603 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 10:47:21.721937 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
I20260928 10:47:21.735885 22665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 3
I20260928 10:47:21.737160 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.736661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:52994)
0928 10:47:21.736903 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.736999 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 10:47:21.738277 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.737859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53006)
0928 10:47:21.738081 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.738162 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260928 10:47:21.738483 22665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
I20260928 10:47:21.790093 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 2
I20260928 10:47:21.791260 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.790773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53010)
0928 10:47:21.790997 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.791096 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 10:47:21.792306 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.791970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53022)
0928 10:47:21.792117 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.792201 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 10:47:21.792493 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
W20260928 10:47:21.867419 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:21.867938 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:21.868098 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:21.876319 22672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:21.876451 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:21.876510 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:21.876590 22672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:21.876658 22672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:21.881705 22665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 4
I20260928 10:47:21.882902 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.882410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53030)
0928 10:47:21.882650 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.882748 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 10:47:21.883972 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.883554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53036)
0928 10:47:21.883737 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.883816 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 10:47:21.884191 22665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
I20260928 10:47:21.887955 22672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:36007,127.8.218.253:43637,127.8.218.252:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab
--principal=kudu/127.8.218.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.252:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.218.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:21.889716 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:21.891202 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 10:47:21.898701 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 3
I20260928 10:47:21.899784 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.899335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53044)
0928 10:47:21.899535 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.899624 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 10:47:21.900899 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:21.900485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53054)
0928 10:47:21.900702 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:21.900786 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260928 10:47:21.901160 22591 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:40681: Network error: Client connection negotiation failed: client connection to 127.8.218.252:40681: connect: Connection refused (error 111)
W20260928 10:47:21.904134 22677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:21.904177 22680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:21.904315 22678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:21.905018 22672 server_base.cc:1034] running on GCE node
I20260928 10:47:21.905900 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:21.907409 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:21.908670 22672 hybrid_clock.cc:648] HybridClock initialized: now 1790592441908590 us; error 57 us; skew 500 ppm
I20260928 10:47:21.909094 22672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:21 dist-test-slave-mn38 krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592441, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:21.912564 22672 init.cc:377] Logged in from keytab as kudu/127.8.218.252@KRBTEST.COM (short username kudu)
I20260928 10:47:21.914237 22672 webserver.cc:460] Webserver started at http://127.8.218.252:35313/ using document root <none> and password file <none>
I20260928 10:47:21.914980 22672 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:21.915109 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:21.915395 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:21.918597 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/data/instance:
uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e"
format_stamp: "Formatted at 2026-09-28 10:47:21 on dist-test-slave-mn38"
server_key: "79f5ef48e0c4590ea03c77191531ca28"
server_key_iv: "7fc2de3edae1f907129a22b0ada9f55d"
server_key_version: "encryptionkey@0"
I20260928 10:47:21.919435 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/wal/instance:
uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e"
format_stamp: "Formatted at 2026-09-28 10:47:21 on dist-test-slave-mn38"
server_key: "79f5ef48e0c4590ea03c77191531ca28"
server_key_iv: "7fc2de3edae1f907129a22b0ada9f55d"
server_key_version: "encryptionkey@0"
I20260928 10:47:21.925386 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:47:21.929252 22687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:21.930811 22672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:47:21.931008 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/wal
uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e"
format_stamp: "Formatted at 2026-09-28 10:47:21 on dist-test-slave-mn38"
server_key: "79f5ef48e0c4590ea03c77191531ca28"
server_key_iv: "7fc2de3edae1f907129a22b0ada9f55d"
server_key_version: "encryptionkey@0"
I20260928 10:47:21.931211 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:21.964294 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:21.968928 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:21.969215 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:21.988413 22672 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.252:40681
I20260928 10:47:21.988517 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.252:40681 every 8 connection(s)
I20260928 10:47:21.989953 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
I20260928 10:47:21.994086 22739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:21.998880 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 22672
I20260928 10:47:21.999099 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790592230404718-9067-0/minicluster-data/master-2/wal/instance
I20260928 10:47:21.999716 9067 external_mini_cluster.cc:1428] Setting key 53dfc562caee73248a165d333f1be002
I20260928 10:47:22.002156 22739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:47:22 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592441, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
Sep 28 10:47:22 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
I20260928 10:47:22.021823 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.005115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.8.218.254:36007)
0928 10:47:22.005669 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.005674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.006334 (+ 660us) server_negotiation.cc:407] Connection header received
0928 10:47:22.006399 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.006403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.006467 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.006556 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.007911 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.008706 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.010417 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.010839 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.010966 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.014720 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.014727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.014730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.014769 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.017751 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.018770 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.018777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.018780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.019007 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.019614 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.019620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.019623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.020005 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.020138 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.021075 (+ 937us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.021353 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":253,"threads_started":1}
I20260928 10:47:22.023123 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.001994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.8.218.252:40681)
0928 10:47:22.002907 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.002921 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.002969 (+ 48us) server_negotiation.cc:407] Connection header received
0928 10:47:22.004791 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.004820 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.005312 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.005896 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.007379 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.008780 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.010282 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.010842 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.011017 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.015177 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.015185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.015199 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.015232 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.018314 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.019237 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.019246 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.019249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.019446 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.019948 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.019952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.019954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.020657 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.020968 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.021358 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.021649 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":671,"spinlock_wait_cycles":44800,"thread_start_us":415,"threads_started":1}
I20260928 10:47:22.023173 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.005084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36007 (local address 127.0.0.1:41514)
0928 10:47:22.006013 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:22.006060 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:22.006238 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:22.006652 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:22.006659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:22.007241 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:22.007722 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.007740 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.008877 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.008881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.010255 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.010270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.011091 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.011099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.011236 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.011948 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:22.011972 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:22.014444 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:22.017927 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.017940 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.017953 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.018591 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.019194 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.019199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.019203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.019469 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.020122 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:22.020131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:22.020650 (+ 519us) client_negotiation.cc:769] Sending connection context
0928 10:47:22.020940 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 10:47:22.021547 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":18,"thread_start_us":526,"threads_started":1}
I20260928 10:47:22.024914 22739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:47:22 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592441, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
Sep 28 10:47:22 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:22.040675 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.025605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:43637 (local address 127.0.0.1:37216)
0928 10:47:22.025785 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:22.025801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:22.025912 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:22.026299 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:22.026303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:22.026814 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:22.027022 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.027035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.028426 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.028430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.029626 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.029646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.030483 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.030489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.030664 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.031448 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:22.031474 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:22.034232 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:22.037992 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.037998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.038002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.038536 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.039007 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.039010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.039013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.039175 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.039824 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:22.039828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:22.040000 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 10:47:22.040116 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 10:47:22.040342 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 10:47:22.040974 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.025784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.8.218.253:43637)
0928 10:47:22.025986 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.025992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.026010 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:47:22.026070 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.026074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.026137 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.026236 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.027182 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.028262 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.029780 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.030290 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.030432 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.034560 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.034566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.034569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.034608 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.037782 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.038694 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.038702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.038705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.038869 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.039302 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.039308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.039311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.039693 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.039847 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.040261 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.040499 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 10:47:22.041911 22739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:47:22.043242 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.026863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.8.218.254:36007)
0928 10:47:22.027030 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.027035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.027216 (+ 181us) server_negotiation.cc:407] Connection header received
0928 10:47:22.027379 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.027382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.027458 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.027532 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.028986 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.029984 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.031544 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.032246 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.032388 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.037249 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.037257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.037262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.037309 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.040364 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.041238 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.041244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.041248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.041395 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.041803 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.041808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.041811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.042128 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.042395 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.042702 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.042932 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 10:47:22 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592441, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
Sep 28 10:47:22 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
I20260928 10:47:22.056694 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.042565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53062)
0928 10:47:22.042728 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:22.042746 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:22.042843 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:22.043311 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:22.043314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:22.043780 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:22.043967 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.043977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.045084 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.045090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.046506 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.046526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.047257 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.047262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.047420 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.048083 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:22.048101 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:22.050831 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:22.054189 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.054194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.054197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.054601 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.054965 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.054970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.054972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.055093 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.055660 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:22.055664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:22.055798 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 10:47:22.055944 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 10:47:22.056273 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260928 10:47:22.056748 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.042783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.8.218.252:40681)
0928 10:47:22.042924 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.042929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.042943 (+ 14us) server_negotiation.cc:407] Connection header received
0928 10:47:22.042982 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.042985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.043035 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.043109 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.044095 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.044944 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.046678 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.047107 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.047239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.051045 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.051050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.051053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.051094 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.054016 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.054721 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.054727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.054729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.054856 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.055189 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.055193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.055195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.055552 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.055684 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.056035 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.056430 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":51}
I20260928 10:47:22.061154 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.045111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.8.218.253:43637)
0928 10:47:22.045291 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.045296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.045426 (+ 130us) server_negotiation.cc:407] Connection header received
0928 10:47:22.045615 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.045620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.045682 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.045754 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.047254 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.048252 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.049956 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.050516 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.050625 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.055198 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.055204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.055207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.055247 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.058122 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.058941 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.058948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.058951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.059126 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.059525 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.059530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.059534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.059880 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.060149 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.060470 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.060715 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:47:22.063190 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e: Bootstrap starting.
I20260928 10:47:22.066512 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:47:22.067116 22739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:22.068455 22739 log.cc:824] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:22.071666 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e: No bootstrap required, opened a new log
I20260928 10:47:22.078433 22591 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 4
I20260928 10:47:22.083391 22739 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.083981 22739 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:22.084138 22739 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c0ec5ef3df4e4589f9fe1421e00a8e, State: Initialized, Role: FOLLOWER
I20260928 10:47:22.085108 22739 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.090941 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.072531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.8.218.254:36007)
0928 10:47:22.072729 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.072736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.073562 (+ 826us) server_negotiation.cc:407] Connection header received
0928 10:47:22.073767 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.073772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.073841 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.073921 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.075713 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.076462 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.078623 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.079389 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.079649 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.083586 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.083594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.083600 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.083650 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.087255 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.088322 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.088330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.088335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.088512 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.089018 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.089024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.089027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.089388 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.089707 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.090310 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.090566 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:47:22.095475 22753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.096118 22753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:47:22.097013 22739 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:22.098112 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.072681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.8.218.253:43637)
0928 10:47:22.072909 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.072916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.078349 (+ 5433us) server_negotiation.cc:407] Connection header received
0928 10:47:22.078640 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.078646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.078735 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.078804 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.081643 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.082780 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.084852 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.085572 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.085682 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.089040 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.089047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.089052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.089099 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.092762 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.095558 (+ 2796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.095566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.095570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.095743 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.096212 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.096217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.096220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.096622 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.096857 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.097485 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.097746 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 10:47:22 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592440, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
I20260928 10:47:22.109599 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.078303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.8.218.252:40681)
0928 10:47:22.078906 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.078912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.092153 (+ 13241us) server_negotiation.cc:407] Connection header received
0928 10:47:22.092426 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.092432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.092537 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.092611 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.095048 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.096198 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.098003 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.098837 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.098933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.102214 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.102219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.102222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.102257 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.105816 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.106791 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.106797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.106800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.107008 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.107599 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.107604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.107608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.108144 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.108403 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.108837 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.109130 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":457,"spinlock_wait_cycles":17792}
I20260928 10:47:22.115115 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.079267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53078)
0928 10:47:22.079637 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:22.079656 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:22.079758 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:22.096449 (+ 16691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:22.096455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:22.096955 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:22.097174 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.097190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.099855 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.099863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.101513 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.101537 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.103682 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.103690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.104140 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.104723 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:22.104749 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:22.107547 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:22.112118 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.112124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.112128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.112571 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.113031 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.113036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.113038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.113192 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.113995 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:22.114001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:22.114207 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 10:47:22.114395 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 10:47:22.114739 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260928 10:47:22.115131 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.080026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.8.218.252:40681)
0928 10:47:22.094683 (+ 14657us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.094689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.094713 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:47:22.094768 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.094774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.094849 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.097357 (+ 2508us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.097692 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.099657 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.102694 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.103485 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.103627 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.107796 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.107807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.107813 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.107878 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.111946 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.112714 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.112720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.112725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.112889 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.113306 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.113312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.113316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.113749 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.113974 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.114500 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.114750 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":427,"server-negotiator.queue_time_us":14500,"spinlock_wait_cycles":768,"thread_start_us":14222,"threads_started":1}
I20260928 10:47:22.121598 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: Bootstrap starting.
W20260928 10:47:22.123410 22765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:47:22.123632 22765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:47:22.116983 22672 master_runner.cc:186] Error getting master registration for 127.8.218.254:36007: OK, instance_id { permanent_uuid: "56e41aad0a2045459eff32202844a210" instance_seqno: 1790592440932391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:47:22.126085 22672 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:47:22.126436 22591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:22.127918 22591 log.cc:824] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:22.131458 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: No bootstrap required, opened a new log
I20260928 10:47:22.143035 22591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.143522 22591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:22.143656 22591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e41aad0a2045459eff32202844a210, State: Initialized, Role: FOLLOWER
I20260928 10:47:22.144398 22591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.145517 22768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.145853 22768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:47:22.146958 22591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 10:47:22.157586 22779 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:47:22.157840 22779 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:47:22.156919 22524 master_runner.cc:186] Error getting master registration for 127.8.218.253:43637: OK, instance_id { permanent_uuid: "e5826f50106545689215c75454135e60" instance_seqno: 1790592441448933 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:47:22.167153 22524 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 10:47:22.172176 22545 tablet.cc:2369] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:22.183475 22665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } attempt: 5
Sep 28 10:47:22 dist-test-slave-mn38 krb5kdc[22511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592441, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
I20260928 10:47:22.201093 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.184794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.8.218.252:40681)
0928 10:47:22.185039 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:22.185045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:22.185489 (+ 444us) server_negotiation.cc:407] Connection header received
0928 10:47:22.185703 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:22.185708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:22.185786 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:22.185871 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:22.186731 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.187895 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.190445 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.190893 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.191012 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.194644 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:22.194649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:22.194652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:22.194691 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:22.198066 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.198957 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.198962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.198964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.199128 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:22.199548 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:22.199552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:22.199555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:22.199870 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:22.200036 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:22.200453 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 10:47:22.200700 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:47:22.201107 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:22.184317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:40681 (local address 127.0.0.1:53080)
0928 10:47:22.185236 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:22.185258 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:22.185431 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:22.185947 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:22.185951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:22.186375 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:22.186541 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.186550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.188952 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.188958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.190294 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:22.190313 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:22.191042 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:22.191048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:22.191277 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:22.191913 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:22.191931 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:22.194297 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:22.198281 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.198286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.198289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.198810 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.199254 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:22.199260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:22.199264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:22.199416 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:22.200010 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:22.200015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:22.200169 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 10:47:22.200303 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 10:47:22.200701 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783}
I20260928 10:47:22.207417 22665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: Bootstrap starting.
I20260928 10:47:22.211470 22665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:22.212600 22665 log.cc:824] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:22.215426 22665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: No bootstrap required, opened a new log
W20260928 10:47:22.221598 22693 tablet.cc:2369] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:22.223829 22665 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.225152 22665 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:22.225306 22665 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5826f50106545689215c75454135e60, State: Initialized, Role: FOLLOWER
I20260928 10:47:22.226099 22665 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.227438 22782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.227903 22782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:47:22.228739 22665 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:22.250144 22598 master_runner.cc:263] No leader master found from master e5826f50106545689215c75454135e60
I20260928 10:47:22.250332 22598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 10:47:22.253868 22793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:47:22.254135 22793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:47:22.363847 22753 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:47:22.364239 22753 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.366246 22753 leader_election.cc:288] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e41aad0a2045459eff32202844a210 (127.8.218.254:36007), e5826f50106545689215c75454135e60 (127.8.218.253:43637)
I20260928 10:47:22.368026 22640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5826f50106545689215c75454135e60" is_pre_election: true
I20260928 10:47:22.368860 22640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c0ec5ef3df4e4589f9fe1421e00a8e in term 0.
I20260928 10:47:22.369962 22691 leader_election.cc:302] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7c0ec5ef3df4e4589f9fe1421e00a8e, e5826f50106545689215c75454135e60; no voters:
I20260928 10:47:22.371709 22566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e41aad0a2045459eff32202844a210" is_pre_election: true
I20260928 10:47:22.372524 22566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c0ec5ef3df4e4589f9fe1421e00a8e in term 0.
I20260928 10:47:22.373129 22753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:47:22.373339 22753 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:47:22.374405 22753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:22.380297 22753 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.387122 22566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e41aad0a2045459eff32202844a210"
I20260928 10:47:22.387425 22566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:22.391335 22566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c0ec5ef3df4e4589f9fe1421e00a8e in term 1.
I20260928 10:47:22.392050 22688 leader_election.cc:302] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e41aad0a2045459eff32202844a210, a7c0ec5ef3df4e4589f9fe1421e00a8e; no voters:
I20260928 10:47:22.393643 22640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5826f50106545689215c75454135e60"
I20260928 10:47:22.393898 22640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:22.398066 22640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c0ec5ef3df4e4589f9fe1421e00a8e in term 1.
I20260928 10:47:22.399132 22753 leader_election.cc:288] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [CANDIDATE]: Term 1 election: Requested vote from peers 56e41aad0a2045459eff32202844a210 (127.8.218.254:36007), e5826f50106545689215c75454135e60 (127.8.218.253:43637)
I20260928 10:47:22.399631 22753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:22.400107 22753 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [term 1 LEADER]: Becoming Leader. State: Replica: a7c0ec5ef3df4e4589f9fe1421e00a8e, State: Running, Role: LEADER
I20260928 10:47:22.401312 22753 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } }
I20260928 10:47:22.424346 22795 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7c0ec5ef3df4e4589f9fe1421e00a8e. Latest consensus state: current_term: 1 leader_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.424940 22795 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: This master's current role is: LEADER
W20260928 10:47:22.437752 22619 tablet.cc:2369] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:22.442176 22797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:47:22.444325 22797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:47:22.481680 22640 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [term 1 FOLLOWER]: Refusing update from remote peer a7c0ec5ef3df4e4589f9fe1421e00a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:22.483079 22795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:22.491024 22566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [term 1 FOLLOWER]: Refusing update from remote peer a7c0ec5ef3df4e4589f9fe1421e00a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:22.492219 22795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:22.523923 22768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7c0ec5ef3df4e4589f9fe1421e00a8e. Latest consensus state: current_term: 1 leader_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.524268 22768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:47:22.530102 22798 mvcc.cc:205] Tried to move back new op lower bound from 7334266644364808192 to 7334266644109025280. Current Snapshot: MvccSnapshot[applied={T|T < 7334266644364808192}]
I20260928 10:47:22.533978 22753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.534436 22753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:22.541574 22797 catalog_manager.cc:1347] Generated new cluster ID: 543c009a21f64d29b23acab3af581357
I20260928 10:47:22.541719 22797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:47:22.547313 22768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.547667 22768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:47:22.548267 22782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7c0ec5ef3df4e4589f9fe1421e00a8e. Latest consensus state: current_term: 1 leader_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.548619 22782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:47:22.556607 22753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.556906 22753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:22.562692 22799 mvcc.cc:205] Tried to move back new op lower bound from 7334266644364808192 to 7334266644109025280. Current Snapshot: MvccSnapshot[applied={T|T < 7334266644364808192}]
I20260928 10:47:22.569101 22797 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:47:22.571596 22797 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:47:22.589206 22782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e41aad0a2045459eff32202844a210" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36007 } } peers { permanent_uuid: "e5826f50106545689215c75454135e60" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 43637 } } peers { permanent_uuid: "a7c0ec5ef3df4e4589f9fe1421e00a8e" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 40681 } } }
I20260928 10:47:22.589534 22782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:47:22.619585 22797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7c0ec5ef3df4e4589f9fe1421e00a8e: Generated new TSK 0
I20260928 10:47:22.620879 22797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:47:23.160069 22779 catalog_manager.cc:1259] Loaded cluster ID: 543c009a21f64d29b23acab3af581357
I20260928 10:47:23.160189 22779 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: loading cluster ID for follower catalog manager: success
I20260928 10:47:23.162815 22779 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: acquiring CA information for follower catalog manager: success
I20260928 10:47:23.164853 22779 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56e41aad0a2045459eff32202844a210: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:47:23.256204 22793 catalog_manager.cc:1259] Loaded cluster ID: 543c009a21f64d29b23acab3af581357
I20260928 10:47:23.256320 22793 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: loading cluster ID for follower catalog manager: success
I20260928 10:47:23.258900 22793 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: acquiring CA information for follower catalog manager: success
I20260928 10:47:23.260886 22793 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5826f50106545689215c75454135e60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:47:23.270723 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 22524
I20260928 10:47:23.272050 22586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:23.382826 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 22524
I20260928 10:47:23.400424 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 22598
I20260928 10:47:23.401670 22660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:23.510249 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 22598
I20260928 10:47:23.527217 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 22672
I20260928 10:47:23.528839 22734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:23.638619 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 22672
2026-09-28T10:47:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4432 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:47:23 dist-test-slave-mn38 krb5kdc[22817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:47:23 dist-test-slave-mn38 krb5kdc[22817](info): set up 2 sockets
Sep 28 10:47:23 dist-test-slave-mn38 krb5kdc[22817](info): commencing operation
krb5kdc: starting...
W20260928 10:47:24.670749 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:47:24 dist-test-slave-mn38 krb5kdc[22817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:47:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:47:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:47:24.905032 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:43621
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:25.239216 22830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:25.239555 22830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:25.239692 22830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:25.247619 22830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:25.247771 22830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:25.247853 22830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:25.247943 22830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:25.248003 22830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:25.258445 22830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:43621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:25.259815 22830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:25.261185 22830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:25.273574 22835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:25.273494 22836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:25.273488 22838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:25.274724 22830 server_base.cc:1034] running on GCE node
I20260928 10:47:25.275558 22830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:25.277083 22830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:25.278304 22830 hybrid_clock.cc:648] HybridClock initialized: now 1790592445278236 us; error 49 us; skew 500 ppm
I20260928 10:47:25.278724 22830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:25 dist-test-slave-mn38 krb5kdc[22817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592445, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:25.281900 22830 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:47:25.283524 22830 webserver.cc:460] Webserver started at http://127.8.218.254:42223/ using document root <none> and password file <none>
I20260928 10:47:25.284216 22830 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:25.284324 22830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:25.284633 22830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:25.287693 22830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3"
format_stamp: "Formatted at 2026-09-28 10:47:25 on dist-test-slave-mn38"
server_key: "1dc88ba9f4f914814f49e249d9d8a1b5"
server_key_iv: "66713bc3e1c8dd70fedbbd986318e32a"
server_key_version: "encryptionkey@0"
I20260928 10:47:25.288447 22830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3"
format_stamp: "Formatted at 2026-09-28 10:47:25 on dist-test-slave-mn38"
server_key: "1dc88ba9f4f914814f49e249d9d8a1b5"
server_key_iv: "66713bc3e1c8dd70fedbbd986318e32a"
server_key_version: "encryptionkey@0"
I20260928 10:47:25.293916 22830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:47:25.297762 22845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:25.299355 22830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:47:25.299554 22830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3"
format_stamp: "Formatted at 2026-09-28 10:47:25 on dist-test-slave-mn38"
server_key: "1dc88ba9f4f914814f49e249d9d8a1b5"
server_key_iv: "66713bc3e1c8dd70fedbbd986318e32a"
server_key_version: "encryptionkey@0"
I20260928 10:47:25.299758 22830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:25.319347 22830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:25.323864 22830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:25.324149 22830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:25.342504 22830 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:43621
I20260928 10:47:25.342523 22896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:43621 every 8 connection(s)
I20260928 10:47:25.343948 22830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:47:25.346796 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 22830
I20260928 10:47:25.347002 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:47:25.347609 9067 external_mini_cluster.cc:1428] Setting key 37e2a183ded33eab6563c863f3f28b9f
I20260928 10:47:25.347939 22897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:25.362561 22897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3: Bootstrap starting.
Sep 28 10:47:25 dist-test-slave-mn38 krb5kdc[22817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:25.366618 22897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:25.368055 22897 log.cc:824] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:25.371811 22897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3: No bootstrap required, opened a new log
I20260928 10:47:25.373682 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:25.349418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.8.218.254:43621)
0928 10:47:25.350267 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:25.350281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:25.350344 (+ 63us) server_negotiation.cc:407] Connection header received
0928 10:47:25.352791 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:25.352827 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:25.353459 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:25.353958 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:25.355383 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:25.357348 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:25.358831 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:25.359974 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:25.360181 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:25.364568 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:25.364582 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:25.364604 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:25.364664 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:25.368569 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:25.369712 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:25.369722 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:25.369731 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:25.369948 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:25.370332 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:25.370338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:25.370341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:25.371102 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:25.371398 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:25.371830 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 10:47:25.372314 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":330,"threads_started":1}
I20260928 10:47:25.378014 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:47:25.380646 22897 raft_consensus.cc:348] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 43621 } }
I20260928 10:47:25.381042 22897 raft_consensus.cc:374] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:25.381181 22897 raft_consensus.cc:729] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a362ddf8aeac49da9b1ecc7cb66eb2c3, State: Initialized, Role: FOLLOWER
I20260928 10:47:25.381812 22897 consensus_queue.cc:260] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 43621 } }
I20260928 10:47:25.382047 22897 raft_consensus.cc:388] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:47:25.382174 22897 raft_consensus.cc:482] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:47:25.382332 22897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:25.384606 22897 raft_consensus.cc:504] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 43621 } }
I20260928 10:47:25.385185 22897 leader_election.cc:302] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a362ddf8aeac49da9b1ecc7cb66eb2c3; no voters:
I20260928 10:47:25.385885 22897 leader_election.cc:288] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:47:25.386027 22902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:25.386747 22902 raft_consensus.cc:686] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [term 1 LEADER]: Becoming Leader. State: Replica: a362ddf8aeac49da9b1ecc7cb66eb2c3, State: Running, Role: LEADER
I20260928 10:47:25.387337 22902 consensus_queue.cc:237] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 43621 } }
I20260928 10:47:25.388155 22897 sys_catalog.cc:564] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:25.390249 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a362ddf8aeac49da9b1ecc7cb66eb2c3. Latest consensus state: current_term: 1 leader_uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 43621 } } }
I20260928 10:47:25.390307 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a362ddf8aeac49da9b1ecc7cb66eb2c3" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 43621 } } }
I20260928 10:47:25.390655 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:25.390664 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:25.392850 22907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:47:25.398604 22907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:47:25.407646 22907 catalog_manager.cc:1347] Generated new cluster ID: a5d4f7e24f9a4001a149eaf05a45f5cf
I20260928 10:47:25.407776 22907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:47:25.418701 22907 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:47:25.420055 22907 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:47:25.433348 22907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a362ddf8aeac49da9b1ecc7cb66eb2c3: Generated new TSK 0
I20260928 10:47:25.434556 22907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:47:25.490813 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 22830
I20260928 10:47:25.492223 22892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:25.589268 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 22830
2026-09-28T10:47:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1946 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:47:25 dist-test-slave-mn38 krb5kdc[22925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:47:25 dist-test-slave-mn38 krb5kdc[22925](info): set up 2 sockets
Sep 28 10:47:25 dist-test-slave-mn38 krb5kdc[22925](info): commencing operation
krb5kdc: starting...
W20260928 10:47:26.623180 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:47:26 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:47:26.869920 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38195
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:27.213164 22938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:27.213563 22938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:27.213708 22938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:27.221730 22938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:27.221885 22938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:27.221983 22938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:27.222064 22938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:27.222123 22938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:27.233091 22938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:27.234542 22938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:27.235918 22938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:27.248832 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:27.248899 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:27.248868 22946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:27.249737 22938 server_base.cc:1034] running on GCE node
I20260928 10:47:27.250613 22938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:27.252138 22938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:27.253424 22938 hybrid_clock.cc:648] HybridClock initialized: now 1790592447253320 us; error 35 us; skew 500 ppm
I20260928 10:47:27.253840 22938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:27 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592447, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:27.262682 22938 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:47:27.264506 22938 webserver.cc:460] Webserver started at http://127.8.218.254:34957/ using document root <none> and password file <none>
I20260928 10:47:27.265266 22938 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:27.265413 22938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:27.265758 22938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:27.268999 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "aa8e21f1603e4cf1beee34987c3d1818"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "6cd4e34777b52c50d17312239c7afd2d"
server_key_iv: "7a71f770932a3f4463eb01b799a46121"
server_key_version: "encryptionkey@0"
I20260928 10:47:27.269888 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "aa8e21f1603e4cf1beee34987c3d1818"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "6cd4e34777b52c50d17312239c7afd2d"
server_key_iv: "7a71f770932a3f4463eb01b799a46121"
server_key_version: "encryptionkey@0"
I20260928 10:47:27.275736 22938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:47:27.279906 22953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:27.281574 22938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 10:47:27.281770 22938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "aa8e21f1603e4cf1beee34987c3d1818"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "6cd4e34777b52c50d17312239c7afd2d"
server_key_iv: "7a71f770932a3f4463eb01b799a46121"
server_key_version: "encryptionkey@0"
I20260928 10:47:27.281986 22938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:27.311713 22938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:27.316720 22938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:27.317026 22938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:27.336149 22938 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:38195
I20260928 10:47:27.336179 23004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:38195 every 8 connection(s)
I20260928 10:47:27.337729 22938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:47:27.341765 23005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:27.343647 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 22938
I20260928 10:47:27.343888 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:47:27.344647 9067 external_mini_cluster.cc:1428] Setting key 46fec96d5d9f067afb593809b650d707
I20260928 10:47:27.355213 23005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Bootstrap starting.
I20260928 10:47:27.359377 23005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Neither blocks nor log segments found. Creating new log.
Sep 28 10:47:27 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:27.360596 23005 log.cc:824] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:27.364365 23005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: No bootstrap required, opened a new log
I20260928 10:47:27.370493 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:27.346980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.8.218.254:38195)
0928 10:47:27.347818 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:27.347834 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:27.347877 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:47:27.349638 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:27.349666 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:27.350123 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:27.350470 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:27.352337 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:27.353894 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:27.355774 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:27.356542 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:27.356692 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:27.361594 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:27.361603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:27.361620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:27.361666 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:27.365548 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:27.366382 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:27.366392 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:27.366403 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:27.366635 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:27.366986 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:27.366992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:27.366995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:27.367775 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:27.368089 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:27.368494 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 10:47:27.368999 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":360,"threads_started":1}
I20260928 10:47:27.375142 23005 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:27.375566 23005 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:27.375651 23005 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8e21f1603e4cf1beee34987c3d1818, State: Initialized, Role: FOLLOWER
I20260928 10:47:27.376190 23005 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:27.376411 23005 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:47:27.376502 23005 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:47:27.376631 23005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:27.378921 23005 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:27.379462 23005 leader_election.cc:302] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8e21f1603e4cf1beee34987c3d1818; no voters:
I20260928 10:47:27.380141 23005 leader_election.cc:288] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:47:27.380338 23010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:27.381506 23010 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 1 LEADER]: Becoming Leader. State: Replica: aa8e21f1603e4cf1beee34987c3d1818, State: Running, Role: LEADER
I20260928 10:47:27.382045 23010 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:27.382568 23005 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:27.385318 23011 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa8e21f1603e4cf1beee34987c3d1818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } } }
I20260928 10:47:27.385349 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8e21f1603e4cf1beee34987c3d1818. Latest consensus state: current_term: 1 leader_uuid: "aa8e21f1603e4cf1beee34987c3d1818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } } }
I20260928 10:47:27.385911 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:27.385913 23011 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:27.390163 23017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:47:27.394567 23017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:47:27.403494 23017 catalog_manager.cc:1347] Generated new cluster ID: 851dfe5f511b47289e798746a4bfe214
I20260928 10:47:27.403678 23017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:47:27.428433 23017 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:47:27.430536 23017 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:47:27.441119 23017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Generated new TSK 0
I20260928 10:47:27.442147 23017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:47:27.509662 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:27.854385 23031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:27.854702 23031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:27.854835 23031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:27.863152 23031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:27.863319 23031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:27.863454 23031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:47:27.874368 23031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:27.875759 23031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:27.877106 23031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:27.890852 23039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:27.890811 23037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:27.890914 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:27.891681 23031 server_base.cc:1034] running on GCE node
I20260928 10:47:27.892376 23031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:27.893374 23031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:27.894639 23031 hybrid_clock.cc:648] HybridClock initialized: now 1790592447894574 us; error 43 us; skew 500 ppm
I20260928 10:47:27.894999 23031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:27 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592447, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:27.898731 23031 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:47:27.900293 23031 webserver.cc:460] Webserver started at http://127.8.218.193:36867/ using document root <none> and password file <none>
I20260928 10:47:27.901036 23031 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:27.901146 23031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:27.901494 23031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:27.904642 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "2776d905327442cdbe31387d604b0ec5"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "dd6b279ee8b90798b72c1b22fe876ec2"
server_key_iv: "7379b777f1389bae1af2b41c4e0fd7bd"
server_key_version: "encryptionkey@0"
I20260928 10:47:27.905412 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "2776d905327442cdbe31387d604b0ec5"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "dd6b279ee8b90798b72c1b22fe876ec2"
server_key_iv: "7379b777f1389bae1af2b41c4e0fd7bd"
server_key_version: "encryptionkey@0"
I20260928 10:47:27.910815 23031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:47:27.914556 23046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:27.915989 23031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:47:27.916146 23031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "2776d905327442cdbe31387d604b0ec5"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "dd6b279ee8b90798b72c1b22fe876ec2"
server_key_iv: "7379b777f1389bae1af2b41c4e0fd7bd"
server_key_version: "encryptionkey@0"
I20260928 10:47:27.916352 23031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:27.944109 23031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:27.948539 23031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:27.948823 23031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:27.949828 23031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:27.951277 23031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:27.951352 23031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:27.951453 23031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:27.951519 23031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:27.983942 23031 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:40975
I20260928 10:47:27.984002 23158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:40975 every 8 connection(s)
I20260928 10:47:27.985275 23031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:47:27.986078 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 23031
I20260928 10:47:27.986284 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:47:27.987135 9067 external_mini_cluster.cc:1428] Setting key f7410db4c2932db29d063108d4ad44e8
Sep 28 10:47:27 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592447, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:28.008550 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:27.988384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:36095 (local address 127.8.218.254:38195)
0928 10:47:27.988993 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:27.988999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:27.990401 (+ 1402us) server_negotiation.cc:407] Connection header received
0928 10:47:27.992942 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:27.992948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:27.993034 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:27.993142 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:27.995455 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:27.996219 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:27.998324 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:27.998744 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:27.998868 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:28.002672 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:28.002678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:28.002681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:28.002718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:28.005188 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:28.006095 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:28.006101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:28.006105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:28.006257 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:28.006744 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:28.006747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:28.006750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:28.007031 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:28.007158 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:28.008109 (+ 951us) server_negotiation.cc:299] Negotiation successful
0928 10:47:28.008287 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":334,"threads_started":1}
I20260928 10:47:28.010116 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:27.989300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:36095)
0928 10:47:27.990209 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:27.990278 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:27.992561 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:27.993471 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:27.993486 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:27.994275 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:27.995159 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:27.995186 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:27.996382 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:27.996387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:27.998146 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:27.998164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:27.998996 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:27.999004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:27.999233 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:27.999885 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:27.999908 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:28.002385 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:28.005337 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:28.005348 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:28.005367 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:28.005966 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:28.006364 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:28.006369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:28.006373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:28.006628 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:28.007157 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:28.007166 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:28.007711 (+ 545us) client_negotiation.cc:769] Sending connection context
0928 10:47:28.008027 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 10:47:28.008617 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:47:28.011893 23159 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38195
I20260928 10:47:28.012208 23159 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:28.012941 23159 heartbeater.cc:511] Master 127.8.218.254:38195 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
I20260928 10:47:28.015352 22970 ts_manager.cc:194] Registered new tserver with Master: 2776d905327442cdbe31387d604b0ec5 (127.8.218.193:40975)
I20260928 10:47:28.017082 22970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:36095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:47:28.041311 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:28.376902 23165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:28.377216 23165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:28.377315 23165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:28.385334 23165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:28.385495 23165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:28.385663 23165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:47:28.397390 23165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:28.398761 23165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:28.400153 23165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:28.413859 23173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:28.413784 23171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:28.414192 23170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:28.414407 23165 server_base.cc:1034] running on GCE node
I20260928 10:47:28.415259 23165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:28.416250 23165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:28.417531 23165 hybrid_clock.cc:648] HybridClock initialized: now 1790592448417456 us; error 54 us; skew 500 ppm
I20260928 10:47:28.417886 23165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:28 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592448, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:28.421921 23165 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:47:28.423825 23165 webserver.cc:460] Webserver started at http://127.8.218.194:45251/ using document root <none> and password file <none>
I20260928 10:47:28.424609 23165 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:28.424737 23165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:28.425053 23165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:28.428169 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "eb30e7eed1334f58b006af1bc3b0a831"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "8f7edefbcbfd2a39eb400c4989df87b9"
server_key_iv: "79e7453ad84d1dd080f3ce7759d1adab"
server_key_version: "encryptionkey@0"
I20260928 10:47:28.428979 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "eb30e7eed1334f58b006af1bc3b0a831"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "8f7edefbcbfd2a39eb400c4989df87b9"
server_key_iv: "79e7453ad84d1dd080f3ce7759d1adab"
server_key_version: "encryptionkey@0"
I20260928 10:47:28.434391 23165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 10:47:28.438000 23180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:28.439419 23165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 10:47:28.439575 23165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "eb30e7eed1334f58b006af1bc3b0a831"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "8f7edefbcbfd2a39eb400c4989df87b9"
server_key_iv: "79e7453ad84d1dd080f3ce7759d1adab"
server_key_version: "encryptionkey@0"
I20260928 10:47:28.439762 23165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:28.463882 23165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:28.468796 23165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:28.469111 23165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:28.470199 23165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:28.471648 23165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:28.471731 23165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:28.471855 23165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:28.471907 23165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:28.503705 23165 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:39439
I20260928 10:47:28.503738 23292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:39439 every 8 connection(s)
I20260928 10:47:28.505025 23165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:47:28.514279 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 23165
I20260928 10:47:28.514472 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:47:28.515027 9067 external_mini_cluster.cc:1428] Setting key a554f4d1e1d70013c16a2663a3f5ad93
Sep 28 10:47:28 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592448, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:28.527241 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:28.507598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:52137 (local address 127.8.218.254:38195)
0928 10:47:28.507823 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:28.507830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:28.509064 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 10:47:28.511245 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:28.511250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:28.511327 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:28.511419 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:28.513019 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:28.513813 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:28.516121 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:28.516780 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:28.516907 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:28.521098 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:28.521104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:28.521108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:28.521146 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:28.523855 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:28.524727 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:28.524733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:28.524737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:28.524888 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:28.525354 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:28.525360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:28.525364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:28.525713 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:28.525860 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:28.526629 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 10:47:28.526887 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":80}
I20260928 10:47:28.528355 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:28.508080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:52137)
0928 10:47:28.508878 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:28.508924 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:28.510941 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:28.511643 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:28.511652 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:28.512203 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:28.512813 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:28.512834 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:28.513998 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:28.514004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:28.515955 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:28.515972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:28.517036 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:28.517046 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:28.517288 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:28.518102 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:28.518136 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:28.520814 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:28.524088 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:28.524097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:28.524112 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:28.524583 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:28.525014 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:28.525018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:28.525020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:28.525229 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:28.525860 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:28.525870 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:28.526246 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 10:47:28.526506 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 10:47:28.527108 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":326,"threads_started":1}
I20260928 10:47:28.530093 23293 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38195
I20260928 10:47:28.530478 23293 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:28.531232 23293 heartbeater.cc:511] Master 127.8.218.254:38195 requested a full tablet report, sending...
I20260928 10:47:28.532905 22970 ts_manager.cc:194] Registered new tserver with Master: eb30e7eed1334f58b006af1bc3b0a831 (127.8.218.194:39439)
I20260928 10:47:28.534157 22970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:52137
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:47:28.565084 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:28.892335 23298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:28.892652 23298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:28.892777 23298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:28.900380 23298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:28.900516 23298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:28.900671 23298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:47:28.910713 23298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:28.912038 23298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:28.913345 23298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:28.926539 23303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:28.926575 23306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:28.926551 23304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:28.927258 23298 server_base.cc:1034] running on GCE node
I20260928 10:47:28.927904 23298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:28.929061 23298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:28.930260 23298 hybrid_clock.cc:648] HybridClock initialized: now 1790592448930212 us; error 43 us; skew 500 ppm
I20260928 10:47:28.930619 23298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:28 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592448, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:28.933718 23298 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:47:28.935299 23298 webserver.cc:460] Webserver started at http://127.8.218.195:38019/ using document root <none> and password file <none>
I20260928 10:47:28.936014 23298 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:28.936131 23298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:28.936467 23298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:28.939524 23298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "6d5f04b6957743c1949ec7477aae5bd1"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "f45484ec465b9e767ad1744af14cf0df"
server_key_iv: "0ae6a8bcfa04bc2f26d3950e1514ad8c"
server_key_version: "encryptionkey@0"
I20260928 10:47:28.940289 23298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "6d5f04b6957743c1949ec7477aae5bd1"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "f45484ec465b9e767ad1744af14cf0df"
server_key_iv: "0ae6a8bcfa04bc2f26d3950e1514ad8c"
server_key_version: "encryptionkey@0"
I20260928 10:47:28.945649 23298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 10:47:28.949179 23313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:28.950614 23298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:47:28.950784 23298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "6d5f04b6957743c1949ec7477aae5bd1"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "f45484ec465b9e767ad1744af14cf0df"
server_key_iv: "0ae6a8bcfa04bc2f26d3950e1514ad8c"
server_key_version: "encryptionkey@0"
I20260928 10:47:28.950992 23298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:28.972420 23298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:28.977556 23298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:28.977877 23298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:28.978933 23298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:28.980612 23298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:28.980690 23298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:28.980800 23298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:28.980847 23298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:29.014977 23298 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:38433
I20260928 10:47:29.015025 23425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:38433 every 8 connection(s)
I20260928 10:47:29.016237 23298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:47:29.016844 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 23298
I20260928 10:47:29.017027 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:47:29.017675 9067 external_mini_cluster.cc:1428] Setting key de7eaec66c71b45c50fb5e60db66daf5
I20260928 10:47:29.020506 23159 heartbeater.cc:503] Master 127.8.218.254:38195 was elected leader, sending a full tablet report...
Sep 28 10:47:29 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592448, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:29.038065 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.019160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:38961 (local address 127.8.218.254:38195)
0928 10:47:29.019449 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.019455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.021114 (+ 1659us) server_negotiation.cc:407] Connection header received
0928 10:47:29.023372 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.023376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.023446 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.023526 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:29.025129 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.025925 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.027506 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.027965 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.028066 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.031816 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:29.031821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:29.031825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:29.031862 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:29.034899 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.035681 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.035686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.035688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.035839 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.036294 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.036297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.036299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.036657 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:29.036782 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.037601 (+ 819us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.037791 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 10:47:29.039105 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.019883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:38961)
0928 10:47:29.020948 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:29.021005 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:29.023036 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:29.023742 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:29.023751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:29.024365 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:29.024962 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.024980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.026070 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.026074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.027347 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.027361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.028189 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.028194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.028427 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.029149 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:29.029177 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:29.031552 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:29.035055 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:29.035063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:29.035079 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:29.035546 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:29.035977 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:29.035981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:29.035984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:29.036194 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:29.036815 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:29.036822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:29.037228 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 10:47:29.037533 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 10:47:29.037990 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":428,"threads_started":1}
I20260928 10:47:29.040418 23426 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38195
I20260928 10:47:29.040750 23426 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:29.041501 23426 heartbeater.cc:511] Master 127.8.218.254:38195 requested a full tablet report, sending...
I20260928 10:47:29.042986 22969 ts_manager.cc:194] Registered new tserver with Master: 6d5f04b6957743c1949ec7477aae5bd1 (127.8.218.195:38433)
I20260928 10:47:29.043884 22969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:38961
I20260928 10:47:29.047251 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:47:29.065327 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.051933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.8.218.254:38195)
0928 10:47:29.052181 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.052187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.052445 (+ 258us) server_negotiation.cc:407] Connection header received
0928 10:47:29.052599 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.052603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.052674 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.052758 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:29.054186 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.055035 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.056604 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.057026 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.057125 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.059568 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:29.059573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:29.059576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:29.059611 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:29.062184 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.062992 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.062997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.062999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.063170 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.063726 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.063731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.063733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.064131 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:29.064408 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.064795 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.064997 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 10:47:29.070075 22969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:47:29.077337 22969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:47:29.112282 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.090921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.8.218.195:38433)
0928 10:47:29.100890 (+ 9969us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.100902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.102949 (+ 2047us) server_negotiation.cc:407] Connection header received
0928 10:47:29.103335 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.103345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.103620 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.103812 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:29.104950 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.106693 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.109846 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.111232 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.111377 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.111598 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.111854 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.112048 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9716,"thread_start_us":9528,"threads_started":1}
I20260928 10:47:29.112332 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.102389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:40975 (local address 127.0.0.1:46804)
0928 10:47:29.103740 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:29.103756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:29.103871 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:29.105078 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:29.105083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:29.105107 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:29.105423 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.105456 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.106813 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.106818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.109109 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.109123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.111234 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.111240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.111500 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.111561 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 10:47:29.111696 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 10:47:29.111922 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"mutex_wait_us":230,"thread_start_us":1090,"threads_started":1}
I20260928 10:47:29.113054 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.102809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.8.218.193:40975)
0928 10:47:29.104478 (+ 1669us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.104488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.104518 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:47:29.104616 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.104625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.104904 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.105081 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:29.105613 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.106635 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.109277 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.110677 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.110827 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.111084 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.112570 (+ 1486us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.112741 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":319,"threads_started":1}
I20260928 10:47:29.113096 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.090405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:38433 (local address 127.0.0.1:35068)
0928 10:47:29.102755 (+ 12350us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:29.102810 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:29.103077 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:29.104094 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:29.104103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:29.104142 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:29.104728 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.104746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.106847 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.106853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.109698 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.109707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.111452 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.111459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.111644 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.111676 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:47:29.112547 (+ 871us) client_negotiation.cc:240] Negotiation successful
0928 10:47:29.112687 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10658,"mutex_wait_us":407,"spinlock_wait_cycles":675328,"thread_start_us":10484,"threads_started":1}
I20260928 10:47:29.114392 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.090476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.8.218.194:39439)
0928 10:47:29.101883 (+ 11407us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.101892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.103322 (+ 1430us) server_negotiation.cc:407] Connection header received
0928 10:47:29.103426 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.103433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.103617 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.103810 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:29.105026 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.106668 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.111714 (+ 5046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.113112 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.113264 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.113492 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.113958 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.114126 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11249,"thread_start_us":11162,"threads_started":1}
I20260928 10:47:29.117255 23361 tablet_service.cc:1467] Processing CreateTablet for tablet cde5b6d0aa5445d7afae514245590312 (DEFAULT_TABLE table=test-table [id=011e07900e0d4c33a79e6e7a8211a7ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:29.117255 23094 tablet_service.cc:1467] Processing CreateTablet for tablet cde5b6d0aa5445d7afae514245590312 (DEFAULT_TABLE table=test-table [id=011e07900e0d4c33a79e6e7a8211a7ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:29.118904 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde5b6d0aa5445d7afae514245590312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:29.118903 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde5b6d0aa5445d7afae514245590312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:29.119359 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.090425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:39439 (local address 127.0.0.1:59644)
0928 10:47:29.102905 (+ 12480us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:29.102924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:29.103084 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:29.104094 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:29.104103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:29.104146 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:29.104791 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.104804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.106847 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.106853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.109669 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.109687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.113596 (+ 3909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.113602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.113765 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.113798 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 10:47:29.113894 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 10:47:29.114038 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12332,"mutex_wait_us":230,"spinlock_wait_cycles":6912,"thread_start_us":12145,"threads_started":1}
I20260928 10:47:29.123091 23228 tablet_service.cc:1467] Processing CreateTablet for tablet cde5b6d0aa5445d7afae514245590312 (DEFAULT_TABLE table=test-table [id=011e07900e0d4c33a79e6e7a8211a7ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:29.124352 23228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde5b6d0aa5445d7afae514245590312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:29.135758 23444 tablet_bootstrap.cc:492] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Bootstrap starting.
I20260928 10:47:29.136502 23445 tablet_bootstrap.cc:492] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Bootstrap starting.
I20260928 10:47:29.136890 23446 tablet_bootstrap.cc:492] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Bootstrap starting.
I20260928 10:47:29.139796 23446 tablet_bootstrap.cc:654] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:29.139858 23444 tablet_bootstrap.cc:654] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:29.140877 23444 log.cc:824] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:29.141036 23446 log.cc:824] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:29.143199 23445 tablet_bootstrap.cc:654] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:29.144054 23446 tablet_bootstrap.cc:492] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: No bootstrap required, opened a new log
I20260928 10:47:29.144122 23444 tablet_bootstrap.cc:492] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: No bootstrap required, opened a new log
I20260928 10:47:29.144379 23446 ts_tablet_manager.cc:1397] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 10:47:29.144397 23444 ts_tablet_manager.cc:1397] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 10:47:29.144500 23445 log.cc:824] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:29.148617 23445 tablet_bootstrap.cc:492] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: No bootstrap required, opened a new log
I20260928 10:47:29.148944 23445 ts_tablet_manager.cc:1397] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20260928 10:47:29.152397 23446 raft_consensus.cc:348] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.152823 23446 raft_consensus.cc:374] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:29.152951 23446 raft_consensus.cc:729] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb30e7eed1334f58b006af1bc3b0a831, State: Initialized, Role: FOLLOWER
I20260928 10:47:29.153646 23446 consensus_queue.cc:260] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.154866 23293 heartbeater.cc:503] Master 127.8.218.254:38195 was elected leader, sending a full tablet report...
I20260928 10:47:29.155685 23446 ts_tablet_manager.cc:1428] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260928 10:47:29.157028 23444 raft_consensus.cc:348] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.157567 23444 raft_consensus.cc:374] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:29.157672 23444 raft_consensus.cc:729] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2776d905327442cdbe31387d604b0ec5, State: Initialized, Role: FOLLOWER
I20260928 10:47:29.158387 23444 consensus_queue.cc:260] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.159982 23445 raft_consensus.cc:348] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.160565 23445 raft_consensus.cc:374] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:29.160732 23445 raft_consensus.cc:729] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5f04b6957743c1949ec7477aae5bd1, State: Initialized, Role: FOLLOWER
I20260928 10:47:29.161216 23444 ts_tablet_manager.cc:1428] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Time spent starting tablet: real 0.017s user 0.006s sys 0.007s
I20260928 10:47:29.161638 23445 consensus_queue.cc:260] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.162946 23426 heartbeater.cc:503] Master 127.8.218.254:38195 was elected leader, sending a full tablet report...
I20260928 10:47:29.163712 23445 ts_tablet_manager.cc:1428] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260928 10:47:29.223110 23450 raft_consensus.cc:482] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:47:29.223603 23450 raft_consensus.cc:504] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.225466 23450 leader_election.cc:288] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d5f04b6957743c1949ec7477aae5bd1 (127.8.218.195:38433), 2776d905327442cdbe31387d604b0ec5 (127.8.218.193:40975)
I20260928 10:47:29.233093 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.225801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:34243 (local address 127.8.218.195:38433)
0928 10:47:29.225991 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.225995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.226652 (+ 657us) server_negotiation.cc:407] Connection header received
0928 10:47:29.226858 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.226864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.226948 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.227058 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:29.227843 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.228623 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.231146 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.232133 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.232327 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.232407 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.232689 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.232830 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 10:47:29.233078 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.225810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:38433 (local address 127.8.218.194:34243)
0928 10:47:29.226534 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:29.226554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:29.226697 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:29.227261 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:29.227266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:29.227302 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:29.227716 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.227728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.228757 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.228762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.230909 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.230930 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.232384 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.232389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.232532 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.232560 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 10:47:29.232641 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 10:47:29.232754 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":30,"thread_start_us":359,"threads_started":1}
I20260928 10:47:29.234025 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.226934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:34259 (local address 127.8.218.193:40975)
0928 10:47:29.227182 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.227188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.227601 (+ 413us) server_negotiation.cc:407] Connection header received
0928 10:47:29.227864 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.227868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.227931 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.228029 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:29.228685 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.229760 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.231870 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.233008 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.233187 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.233263 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.233652 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.233787 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:47:29.234117 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.226658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:40975 (local address 127.8.218.194:34259)
0928 10:47:29.227461 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:29.227479 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:29.227626 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:29.228080 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:29.228085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:29.228111 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:29.228480 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.228493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.229926 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.229932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.231666 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:29.231680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.233267 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.233273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:29.233480 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.233517 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 10:47:29.233599 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 10:47:29.233734 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":1,"thread_start_us":471,"threads_started":1}
I20260928 10:47:29.234269 23381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde5b6d0aa5445d7afae514245590312" candidate_uuid: "eb30e7eed1334f58b006af1bc3b0a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5f04b6957743c1949ec7477aae5bd1" is_pre_election: true
I20260928 10:47:29.234897 23381 raft_consensus.cc:2454] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb30e7eed1334f58b006af1bc3b0a831 in term 0.
I20260928 10:47:29.235261 23114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde5b6d0aa5445d7afae514245590312" candidate_uuid: "eb30e7eed1334f58b006af1bc3b0a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2776d905327442cdbe31387d604b0ec5" is_pre_election: true
I20260928 10:47:29.235608 23181 leader_election.cc:302] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5f04b6957743c1949ec7477aae5bd1, eb30e7eed1334f58b006af1bc3b0a831; no voters:
I20260928 10:47:29.235899 23114 raft_consensus.cc:2454] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb30e7eed1334f58b006af1bc3b0a831 in term 0.
I20260928 10:47:29.236119 23450 raft_consensus.cc:2781] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:47:29.236260 23450 raft_consensus.cc:482] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:47:29.236469 23450 raft_consensus.cc:3037] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Advancing to term 1
W20260928 10:47:29.237916 23160 tablet.cc:2369] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:29.239187 23450 raft_consensus.cc:504] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.239751 23450 leader_election.cc:288] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [CANDIDATE]: Term 1 election: Requested vote from peers 6d5f04b6957743c1949ec7477aae5bd1 (127.8.218.195:38433), 2776d905327442cdbe31387d604b0ec5 (127.8.218.193:40975)
I20260928 10:47:29.240361 23381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde5b6d0aa5445d7afae514245590312" candidate_uuid: "eb30e7eed1334f58b006af1bc3b0a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5f04b6957743c1949ec7477aae5bd1"
I20260928 10:47:29.240423 23114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde5b6d0aa5445d7afae514245590312" candidate_uuid: "eb30e7eed1334f58b006af1bc3b0a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2776d905327442cdbe31387d604b0ec5"
I20260928 10:47:29.240553 23381 raft_consensus.cc:3037] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:29.240634 23114 raft_consensus.cc:3037] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:29.242967 23381 raft_consensus.cc:2454] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb30e7eed1334f58b006af1bc3b0a831 in term 1.
I20260928 10:47:29.243032 23114 raft_consensus.cc:2454] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb30e7eed1334f58b006af1bc3b0a831 in term 1.
I20260928 10:47:29.243484 23181 leader_election.cc:302] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5f04b6957743c1949ec7477aae5bd1, eb30e7eed1334f58b006af1bc3b0a831; no voters:
I20260928 10:47:29.243862 23450 raft_consensus.cc:2781] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:29.244463 23450 raft_consensus.cc:686] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 1 LEADER]: Becoming Leader. State: Replica: eb30e7eed1334f58b006af1bc3b0a831, State: Running, Role: LEADER
I20260928 10:47:29.245074 23450 consensus_queue.cc:237] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:29.251116 22970 catalog_manager.cc:5495] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb30e7eed1334f58b006af1bc3b0a831 (127.8.218.194). New cstate: current_term: 1 leader_uuid: "eb30e7eed1334f58b006af1bc3b0a831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } health_report { overall_health: UNKNOWN } } }
W20260928 10:47:29.257043 23294 tablet.cc:2369] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:47:29.268263 23427 tablet.cc:2369] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:29.273209 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.267149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.8.218.194:39439)
0928 10:47:29.267320 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.267325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.267342 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:47:29.267435 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.267439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.267535 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.267668 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:47:29.268146 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.269227 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.270849 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.271281 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.271448 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.271990 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:47:29.272600 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:47:29.272718 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.272833 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.272918 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:47:29.281677 23114 raft_consensus.cc:1261] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 1 FOLLOWER]: Refusing update from remote peer eb30e7eed1334f58b006af1bc3b0a831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:29.281677 23381 raft_consensus.cc:1261] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 1 FOLLOWER]: Refusing update from remote peer eb30e7eed1334f58b006af1bc3b0a831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:29.282670 23455 consensus_queue.cc:1035] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:29.283114 23450 consensus_queue.cc:1035] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:29.299413 23458 mvcc.cc:205] Tried to move back new op lower bound from 7334266672247255040 to 7334266672113573888. Current Snapshot: MvccSnapshot[applied={T|T < 7334266672247255040}]
I20260928 10:47:29.301080 23459 mvcc.cc:205] Tried to move back new op lower bound from 7334266672247255040 to 7334266672113573888. Current Snapshot: MvccSnapshot[applied={T|T < 7334266672247255040}]
I20260928 10:47:29.317260 22970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:47:29.317647 22970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:47:29.322012 22970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Sending DeleteTablet for 3 replicas of tablet cde5b6d0aa5445d7afae514245590312
I20260928 10:47:29.323385 23361 tablet_service.cc:1514] Processing DeleteTablet for tablet cde5b6d0aa5445d7afae514245590312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:29 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:35068
I20260928 10:47:29.323408 23094 tablet_service.cc:1514] Processing DeleteTablet for tablet cde5b6d0aa5445d7afae514245590312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:29 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:46804
I20260928 10:47:29.323846 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 23031
I20260928 10:47:29.324116 23228 tablet_service.cc:1514] Processing DeleteTablet for tablet cde5b6d0aa5445d7afae514245590312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:29 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:59644
I20260928 10:47:29.324281 23468 tablet_replica.cc:330] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: stopping tablet replica
I20260928 10:47:29.324740 23468 raft_consensus.cc:2229] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:47:29.325462 23468 raft_consensus.cc:2258] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:29.331938 23468 ts_tablet_manager.cc:1905] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:29.336124 23470 tablet_replica.cc:330] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: stopping tablet replica
I20260928 10:47:29.336604 23470 raft_consensus.cc:2229] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:47:29.337507 23470 raft_consensus.cc:2258] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:29.340245 23470 ts_tablet_manager.cc:1905] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:29.349205 23470 ts_tablet_manager.cc:1918] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:29.358908 23468 ts_tablet_manager.cc:1918] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:29.359146 23468 log.cc:1196] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/cde5b6d0aa5445d7afae514245590312
I20260928 10:47:29.359728 23468 ts_tablet_manager.cc:1939] T cde5b6d0aa5445d7afae514245590312 P 6d5f04b6957743c1949ec7477aae5bd1: Deleting consensus metadata
I20260928 10:47:29.361302 23469 tablet_replica.cc:330] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: stopping tablet replica
I20260928 10:47:29.361866 23469 raft_consensus.cc:2229] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:47:29.362000 22954 catalog_manager.cc:4841] TS 6d5f04b6957743c1949ec7477aae5bd1 (127.8.218.195:38433): tablet cde5b6d0aa5445d7afae514245590312 (table test-table [id=011e07900e0d4c33a79e6e7a8211a7ba]) successfully deleted
I20260928 10:47:29.362452 23469 raft_consensus.cc:2258] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:29.362995 23470 log.cc:1196] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/cde5b6d0aa5445d7afae514245590312
I20260928 10:47:29.363690 23470 ts_tablet_manager.cc:1939] T cde5b6d0aa5445d7afae514245590312 P eb30e7eed1334f58b006af1bc3b0a831: Deleting consensus metadata
I20260928 10:47:29.364943 23469 ts_tablet_manager.cc:1905] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:29.365619 22956 catalog_manager.cc:4841] TS eb30e7eed1334f58b006af1bc3b0a831 (127.8.218.194:39439): tablet cde5b6d0aa5445d7afae514245590312 (table test-table [id=011e07900e0d4c33a79e6e7a8211a7ba]) successfully deleted
Sep 28 10:47:29 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:47:29.372749 23469 ts_tablet_manager.cc:1918] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:29.372946 23469 log.cc:1196] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/cde5b6d0aa5445d7afae514245590312
I20260928 10:47:29.373422 23469 ts_tablet_manager.cc:1939] T cde5b6d0aa5445d7afae514245590312 P 2776d905327442cdbe31387d604b0ec5: Deleting consensus metadata
I20260928 10:47:29.375144 22954 catalog_manager.cc:4841] TS 2776d905327442cdbe31387d604b0ec5 (127.8.218.193:40975): tablet cde5b6d0aa5445d7afae514245590312 (table test-table [id=011e07900e0d4c33a79e6e7a8211a7ba]) successfully deleted
I20260928 10:47:29.378152 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.324783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.8.218.193:40975)
0928 10:47:29.324965 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.324972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.355912 (+ 30940us) server_negotiation.cc:407] Connection header received
0928 10:47:29.356970 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.356976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.357066 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.357150 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:29.361157 (+ 4007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.362386 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.364266 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.364887 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.365008 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.370544 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:29.370564 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:29.370584 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:29.370641 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:29.374850 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.375870 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.375877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.375880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.376046 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.376482 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.376486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.376489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.376841 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:29.377086 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.377552 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.377729 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:47:29.378866 23154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:29.515204 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 23031
I20260928 10:47:29.531100 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 23165
Sep 28 10:47:29 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:47:29.547577 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.531853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.8.218.194:39439)
0928 10:47:29.532050 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.532055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.532124 (+ 69us) server_negotiation.cc:407] Connection header received
0928 10:47:29.532342 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.532347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.532423 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.532509 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:29.533916 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.534804 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.536136 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.536606 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.536720 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.541088 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:29.541109 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:29.541125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:29.541171 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:29.544591 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.545357 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.545366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.545370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.545568 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.545961 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.545966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.545970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.546346 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:29.546688 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.546960 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.547186 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:47:29.548231 23288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:29.666936 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 23165
I20260928 10:47:29.682408 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 23298
Sep 28 10:47:29 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:47:29.697984 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:29.683229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.8.218.195:38433)
0928 10:47:29.683428 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:29.683434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:29.683454 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:47:29.683597 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:29.683602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:29.683667 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:29.683751 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:29.685169 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.686022 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.687325 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:29.687767 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:29.687888 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:29.692330 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:29.692347 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:29.692363 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:29.692410 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:29.695182 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.695908 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.695917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.695919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.696069 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:29.696392 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:29.696396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:29.696398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:29.696773 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:29.697012 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:29.697328 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 10:47:29.697657 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:47:29.698633 23421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:29.788369 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 23298
I20260928 10:47:29.802631 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 22938
I20260928 10:47:29.803957 23000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:29.915944 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 22938
I20260928 10:47:29.934145 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38195
--webserver_interface=127.8.218.254
--webserver_port=34957
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:30.283725 23476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:30.284044 23476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:30.284133 23476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:30.291834 23476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:30.292001 23476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:30.292049 23476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:30.292106 23476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:30.292156 23476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:30.302639 23476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=34957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:30.304013 23476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:30.305400 23476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:30.328073 23483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:30.328096 23482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:30.328128 23485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:30.328977 23476 server_base.cc:1034] running on GCE node
I20260928 10:47:30.329926 23476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:30.331647 23476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:30.332947 23476 hybrid_clock.cc:648] HybridClock initialized: now 1790592450332883 us; error 48 us; skew 500 ppm
I20260928 10:47:30.333544 23476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:30 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592450, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:30.338937 23476 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:47:30.340790 23476 webserver.cc:460] Webserver started at http://127.8.218.254:34957/ using document root <none> and password file <none>
I20260928 10:47:30.341598 23476 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:30.341732 23476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:30.348577 23476 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 10:47:30.352015 23492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:30.353696 23476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 10:47:30.353892 23476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "aa8e21f1603e4cf1beee34987c3d1818"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "6cd4e34777b52c50d17312239c7afd2d"
server_key_iv: "7a71f770932a3f4463eb01b799a46121"
server_key_version: "encryptionkey@0"
I20260928 10:47:30.355029 23476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:30.369485 23476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:30.374536 23476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:30.374856 23476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:30.393850 23476 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:38195
I20260928 10:47:30.393869 23543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:38195 every 8 connection(s)
I20260928 10:47:30.395519 23476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:47:30.397209 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 23476
I20260928 10:47:30.398211 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:40975
--local_ip_for_outbound_sockets=127.8.218.193
--tserver_master_addrs=127.8.218.254:38195
--webserver_port=36867
--webserver_interface=127.8.218.193
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:47:30.405252 23544 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:47:30.409263 23544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Bootstrap starting.
I20260928 10:47:30.430513 23544 log.cc:824] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:30.444272 23544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:47:30.444860 23544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Bootstrap complete.
I20260928 10:47:30.455773 23544 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:30.456720 23544 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8e21f1603e4cf1beee34987c3d1818, State: Initialized, Role: FOLLOWER
I20260928 10:47:30.457486 23544 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:30.457767 23544 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:47:30.458062 23544 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:47:30.458267 23544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:47:30.462893 23544 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:30.463506 23544 leader_election.cc:302] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8e21f1603e4cf1beee34987c3d1818; no voters:
I20260928 10:47:30.464147 23544 leader_election.cc:288] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:47:30.464299 23548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:47:30.465746 23548 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 2 LEADER]: Becoming Leader. State: Replica: aa8e21f1603e4cf1beee34987c3d1818, State: Running, Role: LEADER
I20260928 10:47:30.466753 23548 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:30.467195 23544 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:30.469498 23549 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aa8e21f1603e4cf1beee34987c3d1818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } } }
I20260928 10:47:30.469937 23549 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:30.474750 23550 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8e21f1603e4cf1beee34987c3d1818. Latest consensus state: current_term: 2 leader_uuid: "aa8e21f1603e4cf1beee34987c3d1818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } } }
I20260928 10:47:30.475723 23550 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:30.478015 23553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:47:30.487735 23553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:47:30.489573 23553 catalog_manager.cc:1259] Loaded cluster ID: 851dfe5f511b47289e798746a4bfe214
I20260928 10:47:30.489779 23553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:47:30.493278 23553 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 10:47:30.497704 23565 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:47:30.510548 23553 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 10:47:30.520563 23553 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:47:30.522794 23553 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Loaded TSK: 0
I20260928 10:47:30.524907 23553 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:47:30.773020 23546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:30.773326 23546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:30.773458 23546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:30.781302 23546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:30.781427 23546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:30.781590 23546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:47:30.795063 23546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:40975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=36867
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:30.796861 23546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:30.798314 23546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:30.814123 23571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:30.814095 23572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:30.815366 23574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:30.815407 23546 server_base.cc:1034] running on GCE node
I20260928 10:47:30.816236 23546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:30.817585 23546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:30.818845 23546 hybrid_clock.cc:648] HybridClock initialized: now 1790592450818771 us; error 65 us; skew 500 ppm
I20260928 10:47:30.819260 23546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:30 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592450, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:30.823446 23546 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:47:30.826123 23546 webserver.cc:460] Webserver started at http://127.8.218.193:36867/ using document root <none> and password file <none>
I20260928 10:47:30.827085 23546 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:30.827251 23546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:30.834797 23546 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 10:47:30.839243 23581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:30.841331 23546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:47:30.841615 23546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "2776d905327442cdbe31387d604b0ec5"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "dd6b279ee8b90798b72c1b22fe876ec2"
server_key_iv: "7379b777f1389bae1af2b41c4e0fd7bd"
server_key_version: "encryptionkey@0"
I20260928 10:47:30.843070 23546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:30.865578 23546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:30.872040 23546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:30.872364 23546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:30.873665 23546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:30.875461 23546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:30.875586 23546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:30.875682 23546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:30.875756 23546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:30.909029 23546 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:40975
I20260928 10:47:30.909113 23693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:40975 every 8 connection(s)
I20260928 10:47:30.910586 23546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:47:30.914928 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 23546
I20260928 10:47:30.915560 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:39439
--local_ip_for_outbound_sockets=127.8.218.194
--tserver_master_addrs=127.8.218.254:38195
--webserver_port=45251
--webserver_interface=127.8.218.194
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:47:30 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592450, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:30.937249 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:30.913671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:59085 (local address 127.8.218.254:38195)
0928 10:47:30.914270 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:30.914282 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:30.914797 (+ 515us) server_negotiation.cc:407] Connection header received
0928 10:47:30.917002 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:30.917020 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:30.917516 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:30.917902 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:30.919329 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:30.921115 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:30.923214 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:30.924032 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:30.924206 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:30.928835 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:30.928846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:30.928863 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:30.928911 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:30.932019 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:30.933006 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:30.933015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:30.933021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:30.933227 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:30.933826 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:30.933830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:30.933832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:30.934382 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:30.934584 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:30.935454 (+ 870us) server_negotiation.cc:299] Negotiation successful
0928 10:47:30.935884 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
I20260928 10:47:30.937245 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:30.913811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:59085)
0928 10:47:30.914604 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:30.914647 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:30.916398 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:30.918046 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:30.918055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:30.918572 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:30.919103 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:30.919119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:30.921329 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:30.921333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:30.923000 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:30.923021 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:30.924319 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:30.924333 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:30.924620 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:30.925414 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:30.925475 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:30.928455 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:30.932301 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:30.932310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:30.932324 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:30.932818 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:30.933367 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:30.933371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:30.933373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:30.933692 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:30.934546 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:30.934555 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:30.934919 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 10:47:30.935177 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 10:47:30.935834 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":302,"threads_started":1}
I20260928 10:47:30.940689 23694 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38195
I20260928 10:47:30.941062 23694 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:30.942062 23694 heartbeater.cc:511] Master 127.8.218.254:38195 requested a full tablet report, sending...
I20260928 10:47:30.945390 23509 ts_manager.cc:194] Registered new tserver with Master: 2776d905327442cdbe31387d604b0ec5 (127.8.218.193:40975)
I20260928 10:47:30.947898 23509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:59085
W20260928 10:47:31.261982 23698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:31.262298 23698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:31.262399 23698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:31.270090 23698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:31.270217 23698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:31.270341 23698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:47:31.280583 23698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:39439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=45251
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:31.281924 23698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:31.283335 23698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:31.296926 23704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:31.296965 23706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:31.296898 23703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:31.297950 23698 server_base.cc:1034] running on GCE node
I20260928 10:47:31.298664 23698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:31.299647 23698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:31.300899 23698 hybrid_clock.cc:648] HybridClock initialized: now 1790592451300828 us; error 61 us; skew 500 ppm
I20260928 10:47:31.301250 23698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:31 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592451, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:31.304612 23698 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:47:31.306325 23698 webserver.cc:460] Webserver started at http://127.8.218.194:45251/ using document root <none> and password file <none>
I20260928 10:47:31.307037 23698 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:31.307166 23698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:31.312848 23698 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:47:31.315977 23713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:31.317462 23698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:47:31.317636 23698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "eb30e7eed1334f58b006af1bc3b0a831"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "8f7edefbcbfd2a39eb400c4989df87b9"
server_key_iv: "79e7453ad84d1dd080f3ce7759d1adab"
server_key_version: "encryptionkey@0"
I20260928 10:47:31.318662 23698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:31.341053 23698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:31.345683 23698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:31.345952 23698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:31.346968 23698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:31.348410 23698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:31.348517 23698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:31.348644 23698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:31.348714 23698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:31.382943 23698 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:39439
I20260928 10:47:31.382995 23825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:39439 every 8 connection(s)
I20260928 10:47:31.384459 23698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:47:31.391765 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 23698
I20260928 10:47:31.392493 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:38433
--local_ip_for_outbound_sockets=127.8.218.195
--tserver_master_addrs=127.8.218.254:38195
--webserver_port=38019
--webserver_interface=127.8.218.195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:47:31 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592451, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:31.409304 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.387385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:54955 (local address 127.8.218.254:38195)
0928 10:47:31.387632 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:31.387638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:31.389086 (+ 1448us) server_negotiation.cc:407] Connection header received
0928 10:47:31.391092 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:31.391097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:31.391179 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:31.391277 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:31.393747 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.394873 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.397291 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.398142 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.398299 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.402719 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:31.402725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:31.402729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:31.402768 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:31.405746 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:31.406627 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:31.406631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:31.406634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:31.406794 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:31.407353 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:31.407358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:31.407361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:31.407662 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:31.407807 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:31.408635 (+ 828us) server_negotiation.cc:299] Negotiation successful
0928 10:47:31.408938 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 10:47:31.410472 23828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.387867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:54955)
0928 10:47:31.388882 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:31.388929 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:31.390738 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:31.391715 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:31.391728 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:31.392553 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:31.393504 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.393536 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.395125 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.395131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.397076 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.397097 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.398461 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.398469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.398773 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.399695 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:31.399736 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:31.402442 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:31.405939 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:31.405950 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:31.405964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:31.406463 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:31.406921 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:31.406927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:31.406931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:31.407182 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:31.407800 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:31.407809 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:31.408203 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 10:47:31.408476 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 10:47:31.409099 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":365,"threads_started":1}
I20260928 10:47:31.412421 23826 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38195
I20260928 10:47:31.412771 23826 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:31.413645 23826 heartbeater.cc:511] Master 127.8.218.254:38195 requested a full tablet report, sending...
I20260928 10:47:31.415277 23509 ts_manager.cc:194] Registered new tserver with Master: eb30e7eed1334f58b006af1bc3b0a831 (127.8.218.194:39439)
I20260928 10:47:31.416154 23509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:54955
W20260928 10:47:31.735242 23829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:31.735581 23829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:31.735707 23829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:31.743899 23829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:31.744056 23829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:31.744223 23829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:47:31.754482 23829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:38433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=38019
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:31.755829 23829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:31.757206 23829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:31.771312 23834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:31.771387 23837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:31.771276 23835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:31.775252 23829 server_base.cc:1034] running on GCE node
I20260928 10:47:31.776059 23829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:31.777104 23829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:31.778347 23829 hybrid_clock.cc:648] HybridClock initialized: now 1790592451778289 us; error 42 us; skew 500 ppm
I20260928 10:47:31.778704 23829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:31 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592451, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:31.782124 23829 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:47:31.783919 23829 webserver.cc:460] Webserver started at http://127.8.218.195:38019/ using document root <none> and password file <none>
I20260928 10:47:31.784703 23829 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:31.784828 23829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:31.790648 23829 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:47:31.793642 23844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:31.795140 23829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 10:47:31.795332 23829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "6d5f04b6957743c1949ec7477aae5bd1"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "f45484ec465b9e767ad1744af14cf0df"
server_key_iv: "0ae6a8bcfa04bc2f26d3950e1514ad8c"
server_key_version: "encryptionkey@0"
I20260928 10:47:31.796347 23829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:31.814936 23829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:31.821370 23829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:31.821794 23829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:31.823176 23829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:31.825115 23829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:31.825227 23829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:31.825333 23829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:31.825471 23829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 10:47:31.862221 23829 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:38433
I20260928 10:47:31.862462 23956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:38433 every 8 connection(s)
I20260928 10:47:31.864178 23829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:47:31.872853 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 23829
Sep 28 10:47:31 dist-test-slave-mn38 krb5kdc[22925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592451, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:31.890028 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.873961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.8.218.254:38195)
0928 10:47:31.874466 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:31.874472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:31.875164 (+ 692us) server_negotiation.cc:407] Connection header received
0928 10:47:31.875226 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:31.875231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:31.875299 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:31.875389 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:31.877366 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.878498 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.880408 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.881125 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.881223 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.884464 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:31.884469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:31.884473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:31.884509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:31.887364 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:31.888064 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:31.888068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:31.888071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:31.888201 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:31.888565 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:31.888569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:31.888571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:31.888861 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:31.889117 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:31.889420 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 10:47:31.889674 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
I20260928 10:47:31.891170 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.868613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:47095 (local address 127.8.218.254:38195)
0928 10:47:31.868803 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:31.868810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:31.869130 (+ 320us) server_negotiation.cc:407] Connection header received
0928 10:47:31.873633 (+ 4503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:31.873639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:31.873744 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:31.873844 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:31.876219 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.877382 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.880156 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.880781 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.880917 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.885120 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:31.885126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:31.885131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:31.885185 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:31.887993 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:31.888744 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:31.888748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:31.888751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:31.888877 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:31.889301 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:31.889307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:31.889309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:31.889622 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:31.889747 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:31.890641 (+ 894us) server_negotiation.cc:299] Negotiation successful
0928 10:47:31.890852 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":1920}
I20260928 10:47:31.892393 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.868023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:47095)
0928 10:47:31.868938 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:31.868996 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:31.871807 (+ 2811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:31.874177 (+ 2370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:31.874191 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:31.875056 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:31.875999 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.876031 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.877894 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.877900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.880004 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.880023 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.881005 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.881010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.881255 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.882093 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:31.882127 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:31.884856 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:31.888140 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:31.888150 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:31.888171 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:31.888626 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:31.888987 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:31.888992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:31.888996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:31.889193 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:31.889745 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:31.889753 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:31.890237 (+ 484us) client_negotiation.cc:769] Sending connection context
0928 10:47:31.890556 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 10:47:31.891056 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":357,"threads_started":1}
I20260928 10:47:31.893872 23957 heartbeater.cc:347] Connected to a master server at 127.8.218.254:38195
I20260928 10:47:31.894212 23957 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:31.895097 23957 heartbeater.cc:511] Master 127.8.218.254:38195 requested a full tablet report, sending...
I20260928 10:47:31.896667 23509 ts_manager.cc:194] Registered new tserver with Master: 6d5f04b6957743c1949ec7477aae5bd1 (127.8.218.195:38433)
I20260928 10:47:31.897567 23509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:47095
I20260928 10:47:31.905345 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:47:31.912276 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.906406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.8.218.254:38195)
0928 10:47:31.906622 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:31.906629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:31.906931 (+ 302us) server_negotiation.cc:407] Connection header received
0928 10:47:31.907176 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:31.907180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:31.907323 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:31.907518 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:47:31.907917 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.908679 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.910164 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.910591 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.910701 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.911209 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:47:31.911693 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:47:31.911833 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:31.911931 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 10:47:31.912024 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:47:31.913421 23509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:47:31.921052 23509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:47:31.952673 23694 heartbeater.cc:503] Master 127.8.218.254:38195 was elected leader, sending a full tablet report...
I20260928 10:47:31.960310 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.934426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.8.218.194:39439)
0928 10:47:31.949803 (+ 15377us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:31.949813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:31.949853 (+ 40us) server_negotiation.cc:407] Connection header received
0928 10:47:31.951478 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:31.951487 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:31.951708 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:31.951897 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:31.952916 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.954211 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.957666 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.959149 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.959278 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.959496 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:31.959807 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 10:47:31.959985 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15184,"thread_start_us":15782,"threads_started":1}
I20260928 10:47:31.960477 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.934370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:38433 (local address 127.0.0.1:35074)
0928 10:47:31.948791 (+ 14421us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:31.948812 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:31.950795 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:31.951913 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:31.951919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:31.951949 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:31.952277 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.952290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.955817 (+ 3527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.955821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.957495 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.957508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.959475 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.959481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.959641 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.959666 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:47:31.959751 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 10:47:31.959961 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14264,"mutex_wait_us":29,"thread_start_us":13580,"threads_started":1}
I20260928 10:47:31.960497 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.934262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:39439 (local address 127.0.0.1:59668)
0928 10:47:31.947687 (+ 13425us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:31.947762 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:31.951208 (+ 3446us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x61200002cc28. stack: 00007f23a6c6e66e 00007f23a6c6e430 00007f23a5cf55ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f23a6c640fd 00007f23a6c63d1e 00007f23b9b6d3cd 00007f23b9b6d36c 00007f23b9b39b3b 00007f23ad673536 00007f23ad671d9e 00007f23ad6f9dca 00007f23ad6f7f68 00007f23ad72b82f
0928 10:47:31.951253 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:31.952189 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:31.952196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:31.952230 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:31.952666 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.952678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.954436 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.954441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.957483 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.957500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.959331 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.959336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.959547 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.959607 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 10:47:31.959822 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 10:47:31.959961 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13025,"mutex_wait_us":249,"spinlock_wait_cycles":2847872,"thread_start_us":16392,"threads_started":1}
I20260928 10:47:31.961359 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.934606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.8.218.195:38433)
0928 10:47:31.950608 (+ 16002us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:31.950615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:31.950648 (+ 33us) server_negotiation.cc:407] Connection header received
0928 10:47:31.951094 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:31.951105 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:31.951385 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:31.951558 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:31.952473 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.955669 (+ 3196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.957681 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.959141 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.959278 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.959520 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:31.960882 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0928 10:47:31.961046 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15831,"thread_start_us":15439,"threads_started":1}
I20260928 10:47:31.963442 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.949172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:40975 (local address 127.0.0.1:46820)
0928 10:47:31.951240 (+ 2068us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:31.951267 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:31.951461 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:31.952189 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:31.952196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:31.952266 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:31.952899 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.952911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.954670 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.954675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.957483 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:31.957500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.962575 (+ 5075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.962582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:31.962756 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.962788 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:47:31.962904 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 10:47:31.963053 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1924,"mutex_wait_us":159,"spinlock_wait_cycles":4352,"thread_start_us":3324,"threads_started":1}
I20260928 10:47:31.963459 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:31.949346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.8.218.193:40975)
0928 10:47:31.950771 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:31.950780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:31.951360 (+ 580us) server_negotiation.cc:407] Connection header received
0928 10:47:31.951625 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:31.951633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:31.951817 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:31.951949 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:31.953118 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.954537 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.959625 (+ 5088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:31.962068 (+ 2443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:31.962207 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:31.962426 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:31.962969 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 10:47:31.963142 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1251,"thread_start_us":1096,"threads_started":1}
I20260928 10:47:31.964835 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 77d69b01e466421a8723721cc71ff2de (DEFAULT_TABLE table=test-table [id=e59342bb86334e10bea58221ea5d6df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:31.964829 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 77d69b01e466421a8723721cc71ff2de (DEFAULT_TABLE table=test-table [id=e59342bb86334e10bea58221ea5d6df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:31.966080 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d69b01e466421a8723721cc71ff2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:31.966187 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d69b01e466421a8723721cc71ff2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:31.967262 23629 tablet_service.cc:1467] Processing CreateTablet for tablet 77d69b01e466421a8723721cc71ff2de (DEFAULT_TABLE table=test-table [id=e59342bb86334e10bea58221ea5d6df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:31.968634 23629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d69b01e466421a8723721cc71ff2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:31.979792 23973 tablet_bootstrap.cc:492] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Bootstrap starting.
I20260928 10:47:31.984328 23973 tablet_bootstrap.cc:654] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:31.985719 23975 tablet_bootstrap.cc:492] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Bootstrap starting.
I20260928 10:47:31.985764 23973 log.cc:824] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:31.988447 23974 tablet_bootstrap.cc:492] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Bootstrap starting.
I20260928 10:47:31.989871 23973 tablet_bootstrap.cc:492] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: No bootstrap required, opened a new log
I20260928 10:47:31.989831 23975 tablet_bootstrap.cc:654] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:31.990321 23973 ts_tablet_manager.cc:1397] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260928 10:47:31.991380 23975 log.cc:824] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:31.991909 23974 tablet_bootstrap.cc:654] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:31.993219 23974 log.cc:824] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:31.995155 23975 tablet_bootstrap.cc:492] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: No bootstrap required, opened a new log
I20260928 10:47:31.995496 23975 ts_tablet_manager.cc:1397] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 10:47:31.995908 23974 tablet_bootstrap.cc:492] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: No bootstrap required, opened a new log
I20260928 10:47:31.996155 23974 ts_tablet_manager.cc:1397] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 10:47:32.001156 23973 raft_consensus.cc:348] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.001744 23973 raft_consensus.cc:374] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:32.001875 23973 raft_consensus.cc:729] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5f04b6957743c1949ec7477aae5bd1, State: Initialized, Role: FOLLOWER
I20260928 10:47:32.002607 23973 consensus_queue.cc:260] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.003897 23957 heartbeater.cc:503] Master 127.8.218.254:38195 was elected leader, sending a full tablet report...
I20260928 10:47:32.004666 23973 ts_tablet_manager.cc:1428] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 10:47:32.004801 23974 raft_consensus.cc:348] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.005298 23974 raft_consensus.cc:374] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:32.005407 23974 raft_consensus.cc:729] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb30e7eed1334f58b006af1bc3b0a831, State: Initialized, Role: FOLLOWER
I20260928 10:47:32.006108 23974 consensus_queue.cc:260] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.006337 23975 raft_consensus.cc:348] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.006790 23975 raft_consensus.cc:374] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:32.006902 23975 raft_consensus.cc:729] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2776d905327442cdbe31387d604b0ec5, State: Initialized, Role: FOLLOWER
I20260928 10:47:32.007329 23826 heartbeater.cc:503] Master 127.8.218.254:38195 was elected leader, sending a full tablet report...
I20260928 10:47:32.007515 23975 consensus_queue.cc:260] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.007889 23974 ts_tablet_manager.cc:1428] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 10:47:32.009641 23975 ts_tablet_manager.cc:1428] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260928 10:47:32.116043 23958 tablet.cc:2369] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:47:32.136596 23827 tablet.cc:2369] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:47:32.162997 23695 tablet.cc:2369] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:32.300272 23980 raft_consensus.cc:482] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:47:32.300559 23980 raft_consensus.cc:504] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.302433 23980 leader_election.cc:288] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d5f04b6957743c1949ec7477aae5bd1 (127.8.218.195:38433), 2776d905327442cdbe31387d604b0ec5 (127.8.218.193:40975)
I20260928 10:47:32.310981 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:32.302971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:54029 (local address 127.8.218.195:38433)
0928 10:47:32.303137 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:32.303141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:32.303653 (+ 512us) server_negotiation.cc:407] Connection header received
0928 10:47:32.303859 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:32.303865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:32.303954 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:32.304059 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:32.304721 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.305779 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.308387 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.309767 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.309947 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:32.310022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:32.310579 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 10:47:32.310719 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 10:47:32.310997 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:32.303951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:57319 (local address 127.8.218.193:40975)
0928 10:47:32.304163 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:32.304169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:32.304329 (+ 160us) server_negotiation.cc:407] Connection header received
0928 10:47:32.304503 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:32.304506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:32.304565 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:32.304636 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:32.305112 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.306068 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.308495 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.309768 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.309943 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:32.310022 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:32.310579 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 10:47:32.310719 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 10:47:32.311190 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:32.303663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:40975 (local address 127.8.218.194:57319)
0928 10:47:32.304231 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:32.304248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:32.304383 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:32.304672 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:32.304677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:32.304701 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:32.304975 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:32.304981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.306192 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.306197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:32.308324 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:32.308336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.310060 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.310064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:32.310345 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:32.310419 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 10:47:32.310523 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 10:47:32.310708 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":90,"thread_start_us":288,"threads_started":1}
I20260928 10:47:32.311226 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:32.302831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:38433 (local address 127.8.218.194:54029)
0928 10:47:32.303528 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:32.303553 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:32.303698 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:32.304172 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:32.304177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:32.304203 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:32.304584 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:32.304593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.305916 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.305920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:32.308171 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:32.308189 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.310056 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.310064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:32.310349 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:32.310419 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 10:47:32.310523 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 10:47:32.310708 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":96,"spinlock_wait_cycles":6528,"thread_start_us":348,"threads_started":1}
I20260928 10:47:32.312325 23912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d69b01e466421a8723721cc71ff2de" candidate_uuid: "eb30e7eed1334f58b006af1bc3b0a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5f04b6957743c1949ec7477aae5bd1" is_pre_election: true
I20260928 10:47:32.312395 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d69b01e466421a8723721cc71ff2de" candidate_uuid: "eb30e7eed1334f58b006af1bc3b0a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2776d905327442cdbe31387d604b0ec5" is_pre_election: true
I20260928 10:47:32.312935 23912 raft_consensus.cc:2454] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb30e7eed1334f58b006af1bc3b0a831 in term 0.
I20260928 10:47:32.312975 23649 raft_consensus.cc:2454] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb30e7eed1334f58b006af1bc3b0a831 in term 0.
I20260928 10:47:32.313612 23714 leader_election.cc:302] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5f04b6957743c1949ec7477aae5bd1, eb30e7eed1334f58b006af1bc3b0a831; no voters:
I20260928 10:47:32.313994 23980 raft_consensus.cc:2781] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:47:32.314157 23980 raft_consensus.cc:482] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:47:32.314262 23980 raft_consensus.cc:3037] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:32.316399 23980 raft_consensus.cc:504] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.317025 23980 leader_election.cc:288] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [CANDIDATE]: Term 1 election: Requested vote from peers 6d5f04b6957743c1949ec7477aae5bd1 (127.8.218.195:38433), 2776d905327442cdbe31387d604b0ec5 (127.8.218.193:40975)
I20260928 10:47:32.317615 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d69b01e466421a8723721cc71ff2de" candidate_uuid: "eb30e7eed1334f58b006af1bc3b0a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2776d905327442cdbe31387d604b0ec5"
I20260928 10:47:32.317641 23912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d69b01e466421a8723721cc71ff2de" candidate_uuid: "eb30e7eed1334f58b006af1bc3b0a831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5f04b6957743c1949ec7477aae5bd1"
I20260928 10:47:32.317812 23912 raft_consensus.cc:3037] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:32.317816 23649 raft_consensus.cc:3037] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:32.319977 23649 raft_consensus.cc:2454] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb30e7eed1334f58b006af1bc3b0a831 in term 1.
I20260928 10:47:32.320014 23912 raft_consensus.cc:2454] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb30e7eed1334f58b006af1bc3b0a831 in term 1.
I20260928 10:47:32.320413 23714 leader_election.cc:302] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5f04b6957743c1949ec7477aae5bd1, eb30e7eed1334f58b006af1bc3b0a831; no voters:
I20260928 10:47:32.320698 23980 raft_consensus.cc:2781] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:32.321123 23980 raft_consensus.cc:686] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 1 LEADER]: Becoming Leader. State: Replica: eb30e7eed1334f58b006af1bc3b0a831, State: Running, Role: LEADER
I20260928 10:47:32.321732 23980 consensus_queue.cc:237] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } }
I20260928 10:47:32.326860 23507 catalog_manager.cc:5495] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb30e7eed1334f58b006af1bc3b0a831 (127.8.218.194). New cstate: current_term: 1 leader_uuid: "eb30e7eed1334f58b006af1bc3b0a831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb30e7eed1334f58b006af1bc3b0a831" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 } health_report { overall_health: UNKNOWN } } }
I20260928 10:47:32.428818 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:32.422106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.8.218.194:39439)
0928 10:47:32.422453 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:32.422459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:32.422721 (+ 262us) server_negotiation.cc:407] Connection header received
0928 10:47:32.423001 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:32.423006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:32.423104 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:32.423287 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:47:32.423788 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.424597 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.426184 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.426689 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.426797 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:32.427445 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:47:32.427987 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:47:32.428125 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:32.428309 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 10:47:32.428402 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 10:47:32.437316 23912 raft_consensus.cc:1261] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 1 FOLLOWER]: Refusing update from remote peer eb30e7eed1334f58b006af1bc3b0a831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:32.437318 23649 raft_consensus.cc:1261] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 1 FOLLOWER]: Refusing update from remote peer eb30e7eed1334f58b006af1bc3b0a831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:32.438333 23984 consensus_queue.cc:1035] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5f04b6957743c1949ec7477aae5bd1" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 38433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:32.438694 23980 consensus_queue.cc:1035] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2776d905327442cdbe31387d604b0ec5" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 40975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:32.450193 23987 mvcc.cc:205] Tried to move back new op lower bound from 7334266685172883456 to 7334266684713934848. Current Snapshot: MvccSnapshot[applied={T|T < 7334266685172883456}]
I20260928 10:47:32.466234 23507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:47:32.466507 23507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:47:32.469972 23507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Sending DeleteTablet for 3 replicas of tablet 77d69b01e466421a8723721cc71ff2de
I20260928 10:47:32.471380 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d69b01e466421a8723721cc71ff2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:32 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:59668
I20260928 10:47:32.471378 23629 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d69b01e466421a8723721cc71ff2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:32 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:46820
I20260928 10:47:32.471479 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d69b01e466421a8723721cc71ff2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:32 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:35074
I20260928 10:47:32.472306 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 23546
I20260928 10:47:32.483897 23998 tablet_replica.cc:330] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: stopping tablet replica
I20260928 10:47:32.484457 23998 raft_consensus.cc:2229] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:47:32.485046 23998 raft_consensus.cc:2258] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:32.489393 23998 ts_tablet_manager.cc:1905] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:32.497525 23998 ts_tablet_manager.cc:1918] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:32.497745 23998 log.cc:1196] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/77d69b01e466421a8723721cc71ff2de
I20260928 10:47:32.498402 23998 ts_tablet_manager.cc:1939] T 77d69b01e466421a8723721cc71ff2de P 2776d905327442cdbe31387d604b0ec5: Deleting consensus metadata
I20260928 10:47:32.500542 23493 catalog_manager.cc:4841] TS 2776d905327442cdbe31387d604b0ec5 (127.8.218.193:40975): tablet 77d69b01e466421a8723721cc71ff2de (table test-table [id=e59342bb86334e10bea58221ea5d6df4]) successfully deleted
I20260928 10:47:32.501649 23999 tablet_replica.cc:330] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: stopping tablet replica
I20260928 10:47:32.502159 23999 raft_consensus.cc:2229] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:47:32.502707 23999 raft_consensus.cc:2258] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:32.504998 23999 ts_tablet_manager.cc:1905] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:32.505896 24000 tablet_replica.cc:330] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: stopping tablet replica
I20260928 10:47:32.506354 24000 raft_consensus.cc:2229] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:47:32.507035 24000 raft_consensus.cc:2258] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:32.510654 24000 ts_tablet_manager.cc:1905] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:32.512737 23999 ts_tablet_manager.cc:1918] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:32.512946 23999 log.cc:1196] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/77d69b01e466421a8723721cc71ff2de
I20260928 10:47:32.513564 23999 ts_tablet_manager.cc:1939] T 77d69b01e466421a8723721cc71ff2de P 6d5f04b6957743c1949ec7477aae5bd1: Deleting consensus metadata
I20260928 10:47:32.515632 23493 catalog_manager.cc:4841] TS 6d5f04b6957743c1949ec7477aae5bd1 (127.8.218.195:38433): tablet 77d69b01e466421a8723721cc71ff2de (table test-table [id=e59342bb86334e10bea58221ea5d6df4]) successfully deleted
I20260928 10:47:32.519062 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:32.473692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.8.218.193:40975)
0928 10:47:32.473889 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:32.473896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:32.497557 (+ 23661us) server_negotiation.cc:407] Connection header received
0928 10:47:32.498088 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:32.498094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:32.498191 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:32.498270 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:32.500347 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.501404 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.503444 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.504010 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.504112 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:32.511802 (+ 7690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:32.511848 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:32.511868 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:32.511932 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:32.516012 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:32.516827 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:32.516836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:32.516839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:32.516996 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:32.517390 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:32.517395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:32.517397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:32.517830 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:32.518148 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:32.518470 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 10:47:32.518697 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:47:32.519062 24000 ts_tablet_manager.cc:1918] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:32.519246 24000 log.cc:1196] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/77d69b01e466421a8723721cc71ff2de
I20260928 10:47:32.519831 23689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:32.519826 24000 ts_tablet_manager.cc:1939] T 77d69b01e466421a8723721cc71ff2de P eb30e7eed1334f58b006af1bc3b0a831: Deleting consensus metadata
I20260928 10:47:32.521761 23495 catalog_manager.cc:4841] TS eb30e7eed1334f58b006af1bc3b0a831 (127.8.218.194:39439): tablet 77d69b01e466421a8723721cc71ff2de (table test-table [id=e59342bb86334e10bea58221ea5d6df4]) successfully deleted
I20260928 10:47:32.610082 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 23546
I20260928 10:47:32.633086 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 23698
I20260928 10:47:32.648824 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:32.633917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.8.218.194:39439)
0928 10:47:32.634125 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:32.634130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:32.634149 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:47:32.634278 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:32.634284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:32.634375 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:32.634469 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:32.635931 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.636869 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.638686 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.639149 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.639270 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:32.641971 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:32.641985 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:32.642001 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:32.642054 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:32.645506 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:32.646491 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:32.646500 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:32.646502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:32.646674 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:32.647152 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:32.647156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:32.647158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:32.647552 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:32.647825 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:32.648191 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 10:47:32.648474 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:47:32.649685 23821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:32.763535 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 23698
I20260928 10:47:32.779462 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 23829
I20260928 10:47:32.794421 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:32.780152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.8.218.195:38433)
0928 10:47:32.780306 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:32.780310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:32.780344 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:47:32.780499 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:32.780503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:32.780574 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:32.780656 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:32.782137 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.782854 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.784226 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:32.784638 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:32.784734 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:32.787402 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:32.787423 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:32.787444 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:32.787489 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:32.791222 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:32.792136 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:32.792147 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:32.792152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:32.792301 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:32.792690 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:32.792696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:32.792699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:32.793163 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:32.793487 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:32.793841 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 10:47:32.794088 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:47:32.795061 23952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:32.884853 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 23829
I20260928 10:47:32.899063 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 23476
I20260928 10:47:32.900442 23539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:33.012562 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 23476
I20260928 10:47:33.030838 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38195
--webserver_interface=127.8.218.254
--webserver_port=34957
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:33.372965 24006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:33.373314 24006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:33.373504 24006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:33.381119 24006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:33.381268 24006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:33.381353 24006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:33.381429 24006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:33.381522 24006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:33.391858 24006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=34957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:33.393242 24006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:33.394658 24006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:33.406582 24011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:33.406706 24012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:33.406729 24014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:33.407850 24006 server_base.cc:1034] running on GCE node
I20260928 10:47:33.408795 24006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:33.410332 24006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:33.411582 24006 hybrid_clock.cc:648] HybridClock initialized: now 1790592453411520 us; error 51 us; skew 500 ppm
I20260928 10:47:33.411969 24006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:33 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592453, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:33.415385 24006 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:47:33.417342 24006 webserver.cc:460] Webserver started at http://127.8.218.254:34957/ using document root <none> and password file <none>
I20260928 10:47:33.418097 24006 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:33.418226 24006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:33.424082 24006 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:47:33.427340 24021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:33.429075 24006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:47:33.429296 24006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "aa8e21f1603e4cf1beee34987c3d1818"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "6cd4e34777b52c50d17312239c7afd2d"
server_key_iv: "7a71f770932a3f4463eb01b799a46121"
server_key_version: "encryptionkey@0"
I20260928 10:47:33.430456 24006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:33.462544 24006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:33.467303 24006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:33.467592 24006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:33.486467 24006 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:38195
I20260928 10:47:33.486490 24072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:38195 every 8 connection(s)
I20260928 10:47:33.488247 24006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:47:33.494160 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 24006
I20260928 10:47:33.495039 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:40975
--local_ip_for_outbound_sockets=127.8.218.193
--tserver_master_addrs=127.8.218.254:38195
--webserver_port=36867
--webserver_interface=127.8.218.193
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:47:33.495716 24073 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:47:33.499399 24073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Bootstrap starting.
I20260928 10:47:33.527760 24073 log.cc:824] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:33.551034 24073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:47:33.551608 24073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818: Bootstrap complete.
I20260928 10:47:33.560849 24073 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:33.561854 24073 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8e21f1603e4cf1beee34987c3d1818, State: Initialized, Role: FOLLOWER
I20260928 10:47:33.562489 24073 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:33.562723 24073 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:47:33.562834 24073 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:47:33.563004 24073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 2 FOLLOWER]: Advancing to term 3
I20260928 10:47:33.568900 24073 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:33.569538 24073 leader_election.cc:302] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8e21f1603e4cf1beee34987c3d1818; no voters:
I20260928 10:47:33.570305 24073 leader_election.cc:288] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 10:47:33.570468 24077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 10:47:33.571553 24077 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [term 3 LEADER]: Becoming Leader. State: Replica: aa8e21f1603e4cf1beee34987c3d1818, State: Running, Role: LEADER
I20260928 10:47:33.572341 24077 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } }
I20260928 10:47:33.572716 24073 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:33.574801 24079 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8e21f1603e4cf1beee34987c3d1818. Latest consensus state: current_term: 3 leader_uuid: "aa8e21f1603e4cf1beee34987c3d1818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } } }
I20260928 10:47:33.575191 24079 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:33.575749 24078 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aa8e21f1603e4cf1beee34987c3d1818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e21f1603e4cf1beee34987c3d1818" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38195 } } }
I20260928 10:47:33.576020 24078 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8e21f1603e4cf1beee34987c3d1818 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:33.586169 24083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:47:33.596968 24083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:47:33.598917 24083 catalog_manager.cc:1259] Loaded cluster ID: 851dfe5f511b47289e798746a4bfe214
I20260928 10:47:33.599169 24083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 10:47:33.601644 24083 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f43b11d59ad google::LogMessage::Fail() at ??:0
@ 0x7f43b11d888a google::LogMessage::SendToLog() at ??:0
@ 0x7f43b11d54de google::LogMessage::Flush() at ??:0
@ 0x7f43b11d6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f43c6c08512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f43c6c05976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f43c6c8d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f43c6c8d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f43b3e21345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f43b3e2491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f43b3e24799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f43b3dffeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f43b70d96db start_thread at ??:0
@ 0x7f43b006f61f clone at ??:0
W20260928 10:47:33.944499 24075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:33.944770 24075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:33.944892 24075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:33.952518 24075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:33.952638 24075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:33.952788 24075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:47:33.962662 24075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:40975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=36867
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:33.964000 24075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:33.965353 24075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:33.978479 24114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:33.978457 24115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:33.978533 24117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:33.979331 24075 server_base.cc:1034] running on GCE node
I20260928 10:47:33.979951 24075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:33.980922 24075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:33.982132 24075 hybrid_clock.cc:648] HybridClock initialized: now 1790592453982077 us; error 41 us; skew 500 ppm
I20260928 10:47:33.982468 24075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:33 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592453, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:33.985736 24075 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:47:33.987385 24075 webserver.cc:460] Webserver started at http://127.8.218.193:36867/ using document root <none> and password file <none>
I20260928 10:47:33.988116 24075 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:33.988245 24075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:33.994303 24075 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:47:33.997283 24124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:33.998842 24075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:47:33.999019 24075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "2776d905327442cdbe31387d604b0ec5"
format_stamp: "Formatted at 2026-09-28 10:47:27 on dist-test-slave-mn38"
server_key: "dd6b279ee8b90798b72c1b22fe876ec2"
server_key_iv: "7379b777f1389bae1af2b41c4e0fd7bd"
server_key_version: "encryptionkey@0"
I20260928 10:47:34.000020 24075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:34.025646 24075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:34.030565 24075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:34.030860 24075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:34.031963 24075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:34.033509 24075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:34.033620 24075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:34.033730 24075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:34.033799 24075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:34.065559 24075 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:40975
I20260928 10:47:34.065615 24236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:40975 every 8 connection(s)
I20260928 10:47:34.066973 24075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:47:34.071959 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.070106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:57591)
0928 10:47:34.070849 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.071099 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":273,"threads_started":1}
W20260928 10:47:34.072628 24237 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:38195 (0 consecutive failures): Network error: Failed to ping master at 127.8.218.254:38195: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
I20260928 10:47:34.073551 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.073173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:43043)
0928 10:47:34.073325 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.073394 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 10:47:34.073661 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 24075
I20260928 10:47:34.074246 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:39439
--local_ip_for_outbound_sockets=127.8.218.194
--tserver_master_addrs=127.8.218.254:38195
--webserver_port=45251
--webserver_interface=127.8.218.194
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:47:34.074386 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.074046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:58601)
0928 10:47:34.074181 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.074269 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 10:47:34.074616 24237 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:47:34.401770 24240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:34.402082 24240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:34.402221 24240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:34.409770 24240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:34.409909 24240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:34.410064 24240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:47:34.420095 24240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:39439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=45251
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:34.421502 24240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:34.422930 24240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:34.438704 24248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:34.438766 24245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:34.438738 24246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:34.439545 24240 server_base.cc:1034] running on GCE node
I20260928 10:47:34.440371 24240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:34.441583 24240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:34.442875 24240 hybrid_clock.cc:648] HybridClock initialized: now 1790592454442763 us; error 63 us; skew 500 ppm
I20260928 10:47:34.443271 24240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:34 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592454, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:34.446825 24240 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:47:34.448434 24240 webserver.cc:460] Webserver started at http://127.8.218.194:45251/ using document root <none> and password file <none>
I20260928 10:47:34.449203 24240 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:34.449321 24240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:34.455215 24240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:47:34.458328 24255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:34.459935 24240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:47:34.460170 24240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "eb30e7eed1334f58b006af1bc3b0a831"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "8f7edefbcbfd2a39eb400c4989df87b9"
server_key_iv: "79e7453ad84d1dd080f3ce7759d1adab"
server_key_version: "encryptionkey@0"
I20260928 10:47:34.461220 24240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:34.481667 24240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:34.486202 24240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:34.486471 24240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:34.487529 24240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:34.488960 24240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:34.489058 24240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:34.489162 24240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:34.489226 24240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:34.522060 24240 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:39439
I20260928 10:47:34.522135 24367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:39439 every 8 connection(s)
I20260928 10:47:34.523571 24240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:47:34.526466 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 24240
I20260928 10:47:34.527117 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:38433
--local_ip_for_outbound_sockets=127.8.218.195
--tserver_master_addrs=127.8.218.254:38195
--webserver_port=38019
--webserver_interface=127.8.218.195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:47:34.528952 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.526815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:46447)
0928 10:47:34.527685 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.527996 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":358,"threads_started":1}
W20260928 10:47:34.529717 24368 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:38195 (0 consecutive failures): Network error: Failed to ping master at 127.8.218.254:38195: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
I20260928 10:47:34.530648 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.530300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:34175)
0928 10:47:34.530467 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.530540 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 10:47:34.531471 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.531147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:60041)
0928 10:47:34.531282 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.531361 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 10:47:34.531706 24368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 10:47:34.868654 24371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:34.868964 24371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:34.869087 24371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:34.876590 24371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:34.876724 24371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:34.876880 24371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:47:34.887385 24371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:38433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=38019
--tserver_master_addrs=127.8.218.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:34.888797 24371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:34.890393 24371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:34.906445 24377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:34.906462 24376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:34.906608 24379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:34.907864 24371 server_base.cc:1034] running on GCE node
I20260928 10:47:34.908586 24371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:34.909641 24371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:34.910892 24371 hybrid_clock.cc:648] HybridClock initialized: now 1790592454910814 us; error 60 us; skew 500 ppm
I20260928 10:47:34.911247 24371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:34 dist-test-slave-mn38 krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592454, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:34.914711 24371 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:47:34.916250 24371 webserver.cc:460] Webserver started at http://127.8.218.195:38019/ using document root <none> and password file <none>
I20260928 10:47:34.916973 24371 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:34.917093 24371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:34.922930 24371 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:47:34.926337 24386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:34.927943 24371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:47:34.928148 24371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "6d5f04b6957743c1949ec7477aae5bd1"
format_stamp: "Formatted at 2026-09-28 10:47:28 on dist-test-slave-mn38"
server_key: "f45484ec465b9e767ad1744af14cf0df"
server_key_iv: "0ae6a8bcfa04bc2f26d3950e1514ad8c"
server_key_version: "encryptionkey@0"
I20260928 10:47:34.929214 24371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:34.948225 24371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:34.952630 24371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:34.952888 24371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:34.953941 24371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:34.955294 24371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:34.955382 24371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:34.955492 24371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:34.955564 24371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:34.986480 24371 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:38433
I20260928 10:47:34.986538 24498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:38433 every 8 connection(s)
I20260928 10:47:34.987954 24371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:47:34.990495 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 24371
I20260928 10:47:34.993280 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.991117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:57847)
0928 10:47:34.992056 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.992415 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":358,"threads_started":1}
W20260928 10:47:34.993937 24499 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:38195 (0 consecutive failures): Network error: Failed to ping master at 127.8.218.254:38195: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
I20260928 10:47:34.994817 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.994460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:48023)
0928 10:47:34.994626 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.994703 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 10:47:34.995747 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:34.995418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:41033)
0928 10:47:34.995555 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:34.995633 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 10:47:34.995994 24499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 10:47:35.076274 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:35.075331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:51419)
0928 10:47:35.075927 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:35.076026 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":285,"threads_started":1}
I20260928 10:47:35.533377 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:35.532400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:56093)
0928 10:47:35.532986 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:35.533106 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":262,"threads_started":1}
I20260928 10:47:35.997664 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:35.996668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:38287)
0928 10:47:35.997196 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:35.997383 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":257,"threads_started":1}
I20260928 10:47:36.078277 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:36.077329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:38293)
0928 10:47:36.077952 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:36.078074 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":274,"threads_started":1}
I20260928 10:47:36.535223 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:36.534468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:58583)
0928 10:47:36.534966 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:36.535054 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":225,"threads_started":1}
I20260928 10:47:36.999482 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:36.998696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:50841)
0928 10:47:36.999180 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:36.999311 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":230,"threads_started":1}
I20260928 10:47:37.080010 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:37.079259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:60465)
0928 10:47:37.079715 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:37.079828 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":231,"threads_started":1}
I20260928 10:47:37.537001 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:37.536243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:42733)
0928 10:47:37.536728 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:37.536813 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260928 10:47:38.001274 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:38.000458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:52323)
0928 10:47:38.000961 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:38.001062 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260928 10:47:38.081671 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:38.080900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:41485)
0928 10:47:38.081315 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:38.081467 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":202,"threads_started":1}
I20260928 10:47:38.538786 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:38.538018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:48913)
0928 10:47:38.538500 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:38.538592 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":224,"threads_started":1}
I20260928 10:47:39.003080 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:39.002211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:45285)
0928 10:47:39.002760 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:39.002855 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":255,"threads_started":1}
I20260928 10:47:39.083549 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:39.082692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:35945)
0928 10:47:39.083241 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:39.083370 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":259,"threads_started":1}
I20260928 10:47:39.540805 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:39.539814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:37863)
0928 10:47:39.540374 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:39.540473 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":300,"threads_started":1}
I20260928 10:47:40.005043 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:40.004188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:37397)
0928 10:47:40.004782 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:40.004871 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":347,"threads_started":1}
I20260928 10:47:40.085387 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:40.084602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:56187)
0928 10:47:40.085096 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:40.085208 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260928 10:47:40.542414 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:40.541706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:39541)
0928 10:47:40.542133 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:40.542226 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260928 10:47:41.006666 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:41.005914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:45129)
0928 10:47:41.006402 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:41.006502 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":252,"threads_started":1}
I20260928 10:47:41.087272 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:41.086471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:48849)
0928 10:47:41.086965 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:41.087054 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":198,"threads_started":1}
I20260928 10:47:41.544319 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:41.543417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:50605)
0928 10:47:41.543997 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:41.544098 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":254,"threads_started":1}
I20260928 10:47:42.008409 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:42.007570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:35713)
0928 10:47:42.008098 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:42.008185 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":246,"threads_started":1}
I20260928 10:47:42.089030 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:42.088275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:35479)
0928 10:47:42.088766 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:42.088865 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":232,"threads_started":1}
I20260928 10:47:42.546136 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:42.545368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:38651)
0928 10:47:42.545875 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:42.545967 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260928 10:47:43.010018 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:43.009294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:58673)
0928 10:47:43.009743 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:43.009825 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":240,"threads_started":1}
I20260928 10:47:43.090763 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:43.090007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:43719)
0928 10:47:43.090491 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:43.090580 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":232,"threads_started":1}
I20260928 10:47:43.547973 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:43.547198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:49919)
0928 10:47:43.547695 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:43.547783 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":239,"threads_started":1}
I20260928 10:47:44.011860 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:44.010978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:54991)
0928 10:47:44.011552 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:44.011646 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":298,"threads_started":1}
I20260928 10:47:44.092630 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:44.091813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:48855)
0928 10:47:44.092336 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:44.092433 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":286,"threads_started":1}
I20260928 10:47:44.549872 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:44.549008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:36979)
0928 10:47:44.549546 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:44.549645 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":275,"threads_started":1}
I20260928 10:47:45.014046 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:45.013000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:47003)
0928 10:47:45.013769 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:45.013870 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":545,"threads_started":1}
I20260928 10:47:45.094750 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:45.093883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:51989)
0928 10:47:45.094429 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:45.094555 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":256,"threads_started":1}
I20260928 10:47:45.551592 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:45.550812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:40527)
0928 10:47:45.551317 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:45.551411 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":259,"threads_started":1}
I20260928 10:47:46.015987 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:46.015146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:59391)
0928 10:47:46.015660 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:46.015784 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":238,"threads_started":1}
I20260928 10:47:46.096550 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:46.095779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:39109)
0928 10:47:46.096295 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:46.096388 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":213,"threads_started":1}
I20260928 10:47:46.553342 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:46.552553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:43583)
0928 10:47:46.553033 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:46.553124 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":239,"threads_started":1}
I20260928 10:47:47.018103 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:47.017115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:50677)
0928 10:47:47.017806 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:47.017921 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":326,"threads_started":1}
I20260928 10:47:47.098807 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:47.097859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:46193)
0928 10:47:47.098502 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:47.098612 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":302,"threads_started":1}
I20260928 10:47:47.555112 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:47.554355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:53799)
0928 10:47:47.554808 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:47.554929 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":215,"threads_started":1}
I20260928 10:47:48.019971 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:48.019141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:38183)
0928 10:47:48.019673 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:48.019782 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":268,"threads_started":1}
I20260928 10:47:48.100708 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:48.099894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:35185)
0928 10:47:48.100410 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:48.100511 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":271,"threads_started":1}
I20260928 10:47:48.556880 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:48.556093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:46797)
0928 10:47:48.556599 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:48.556690 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
I20260928 10:47:49.021994 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:49.021053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:43377)
0928 10:47:49.021717 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:49.021818 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":297,"threads_started":1}
I20260928 10:47:49.102610 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:49.101845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:44493)
0928 10:47:49.102292 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:49.102432 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":212,"threads_started":1}
I20260928 10:47:49.558929 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:49.558112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.194:48161)
0928 10:47:49.558642 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:49.558745 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":265,"threads_started":1}
I20260928 10:47:49.995061 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 24075
I20260928 10:47:50.021602 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:49.996352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.8.218.193:40975)
0928 10:47:49.997069 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:49.997083 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:49.997124 (+ 41us) server_negotiation.cc:407] Connection header received
0928 10:47:49.998833 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:49.998844 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:49.999317 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:49.999673 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:50.001076 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:50.002491 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:50.003819 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:50.004650 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:50.004802 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:50.014327 (+ 9525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:50.014337 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:50.014355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:50.014407 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:50.017606 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:50.018522 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:50.018531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:50.018539 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:50.018760 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:50.019197 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:50.019202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:50.019204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:50.019909 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:50.020188 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:50.020767 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 10:47:50.021244 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":414,"threads_started":1}
I20260928 10:47:50.023265 24232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:50.024034 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:50.023125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.195:44839)
0928 10:47:50.023656 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:50.023791 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":239,"threads_started":1}
I20260928 10:47:50.111304 24550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:50.110106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38195 (local address 127.8.218.193:48685)
0928 10:47:50.110747 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:50.110977 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":266,"threads_started":1}
I20260928 10:47:50.111301 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 24075
I20260928 10:47:50.126876 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 24240
I20260928 10:47:50.147123 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:50.127939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.8.218.194:39439)
0928 10:47:50.128553 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:50.128587 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:50.128629 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:47:50.130252 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:50.130262 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:50.130707 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:50.131036 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:50.132467 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:50.133848 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:50.135186 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:50.135902 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:50.136041 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:50.138705 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:50.138713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:50.138732 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:50.138777 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:50.142710 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:50.143776 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:50.143787 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:50.143797 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:50.144051 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:50.144542 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:50.144548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:50.144552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:50.145350 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:50.145697 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:50.146190 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 10:47:50.146704 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":281,"threads_started":1}
I20260928 10:47:50.149082 24363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:50.234546 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 24240
I20260928 10:47:50.249758 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 24371
I20260928 10:47:50.268810 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:50.250830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.8.218.195:38433)
0928 10:47:50.251443 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:50.251457 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:50.251502 (+ 45us) server_negotiation.cc:407] Connection header received
0928 10:47:50.253206 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:50.253216 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:50.253708 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:50.254049 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:50.255614 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:50.256891 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:50.258243 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:50.258962 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:50.259161 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:50.262049 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:50.262057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:50.262076 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:50.262131 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:50.265073 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:50.265993 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:50.266002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:50.266010 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:50.266216 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:50.266601 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:50.266604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:50.266607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:50.267185 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:50.267484 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:50.267965 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 10:47:50.268483 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":291,"threads_started":1}
I20260928 10:47:50.270408 24494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:50.355114 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 24371
2026-09-28T10:47:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24767 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:47:50 dist-test-slave-mn38 krb5kdc[24557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:47:50 dist-test-slave-mn38 krb5kdc[24557](info): set up 2 sockets
Sep 28 10:47:50 dist-test-slave-mn38 krb5kdc[24557](info): commencing operation
krb5kdc: starting...
W20260928 10:47:51.382751 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:47:51 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:47:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:47:51.618801 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45853
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:51.974223 24570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:51.974638 24570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:51.974838 24570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:51.984020 24570 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 10:47:51.984148 24570 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 10:47:51.984193 24570 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 10:47:51.984277 24570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:47:51.984354 24570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:51.984437 24570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:51.984503 24570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:47:51.984577 24570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:47:51.995270 24570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:51.996600 24570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:51.998028 24570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:52.010550 24575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:52.010674 24576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:52.010754 24578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:52.011106 24570 server_base.cc:1034] running on GCE node
I20260928 10:47:52.012250 24570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:52.013862 24570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:52.015132 24570 hybrid_clock.cc:648] HybridClock initialized: now 1790592472015072 us; error 39 us; skew 500 ppm
I20260928 10:47:52.015502 24570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:52 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592472, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:52.019107 24570 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:47:52.020774 24570 webserver.cc:460] Webserver started at http://127.8.218.254:41433/ using document root <none> and password file <none>
I20260928 10:47:52.021590 24570 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:52.021710 24570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:52.022084 24570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:52.025799 24570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "01e389a5846a4df693105dfc43533618"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "79a46c7f273a8193cfe6bf951539ae75"
server_key_iv: "2d7b46b188e9bb1ec0afa3a7d310c19b"
server_key_version: "encryptionkey@0"
I20260928 10:47:52.026741 24570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "01e389a5846a4df693105dfc43533618"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "79a46c7f273a8193cfe6bf951539ae75"
server_key_iv: "2d7b46b188e9bb1ec0afa3a7d310c19b"
server_key_version: "encryptionkey@0"
I20260928 10:47:52.032697 24570 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:47:52.036984 24585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:52.038636 24570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:47:52.038843 24570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "01e389a5846a4df693105dfc43533618"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "79a46c7f273a8193cfe6bf951539ae75"
server_key_iv: "2d7b46b188e9bb1ec0afa3a7d310c19b"
server_key_version: "encryptionkey@0"
I20260928 10:47:52.039158 24570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:52.063783 24570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:52.068421 24570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:52.068706 24570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:52.088196 24570 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:45853
I20260928 10:47:52.088217 24636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:45853 every 8 connection(s)
I20260928 10:47:52.089697 24570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:47:52.092271 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 24570
I20260928 10:47:52.092475 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:47:52.093146 9067 external_mini_cluster.cc:1428] Setting key 538e46550d10abb9e5cc95bf3f13845f
I20260928 10:47:52.094220 24637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:47:52 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:52.109236 24637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Bootstrap starting.
I20260928 10:47:52.113313 24637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:52.114845 24637 log.cc:824] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:52.118844 24637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: No bootstrap required, opened a new log
I20260928 10:47:52.119984 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:52.095577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.8.218.254:45853)
0928 10:47:52.096478 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:52.096505 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:52.096560 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:47:52.098468 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:52.098511 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:52.099147 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:52.099703 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:52.101278 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:52.103419 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:52.105014 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:52.106111 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:52.106321 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:52.110383 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:52.110396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:52.110416 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:52.110469 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:52.114545 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:52.115577 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:52.115585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:52.115593 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:52.115792 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:52.116412 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:52.116418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:52.116421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:52.117185 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:52.117541 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:52.117980 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 10:47:52.118480 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":357,"threads_started":1}
I20260928 10:47:52.127794 24637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:47:52.128291 24637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:52.128402 24637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e389a5846a4df693105dfc43533618, State: Initialized, Role: FOLLOWER
I20260928 10:47:52.129046 24637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:47:52.129289 24637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:47:52.129465 24637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:47:52.129648 24637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:52.132273 24637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:47:52.133020 24637 leader_election.cc:302] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e389a5846a4df693105dfc43533618; no voters:
I20260928 10:47:52.133708 24637 leader_election.cc:288] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:47:52.133848 24642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:52.134498 24642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 1 LEADER]: Becoming Leader. State: Replica: 01e389a5846a4df693105dfc43533618, State: Running, Role: LEADER
I20260928 10:47:52.135092 24642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:47:52.135885 24637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:47:52.137781 24644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e389a5846a4df693105dfc43533618. Latest consensus state: current_term: 1 leader_uuid: "01e389a5846a4df693105dfc43533618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } } }
I20260928 10:47:52.138157 24644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:52.138877 24643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01e389a5846a4df693105dfc43533618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } } }
I20260928 10:47:52.139283 24643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: This master's current role is: LEADER
I20260928 10:47:52.142359 24648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:47:52.148939 24648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:47:52.158746 24648 catalog_manager.cc:1347] Generated new cluster ID: 981eb65b3c5443ca8b56586bff8238f3
I20260928 10:47:52.158882 24648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:47:52.171023 24648 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:47:52.173111 24648 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:47:52.183856 24648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Generated new TSK 0
I20260928 10:47:52.185115 24648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:47:52.250097 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45853
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:52.606812 24663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:52.607125 24663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:52.607270 24663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:52.614881 24663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:52.615026 24663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:52.615207 24663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:47:52.625556 24663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:52.626991 24663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:52.628386 24663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:52.642656 24669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:52.642599 24668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:52.642685 24671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:52.643589 24663 server_base.cc:1034] running on GCE node
I20260928 10:47:52.644328 24663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:52.645483 24663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:52.646767 24663 hybrid_clock.cc:648] HybridClock initialized: now 1790592472646710 us; error 52 us; skew 500 ppm
I20260928 10:47:52.647253 24663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:52 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592472, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:52.651329 24663 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:47:52.652995 24663 webserver.cc:460] Webserver started at http://127.8.218.193:44109/ using document root <none> and password file <none>
I20260928 10:47:52.653852 24663 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:52.653968 24663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:52.654254 24663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:52.657325 24663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "17cc5096295841d0ac39346423fff035"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "d37ed5c5e80e3d4b1df1eb0843a01cf9"
server_key_iv: "c8153034d27ed218230809f9470d4665"
server_key_version: "encryptionkey@0"
I20260928 10:47:52.658228 24663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "17cc5096295841d0ac39346423fff035"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "d37ed5c5e80e3d4b1df1eb0843a01cf9"
server_key_iv: "c8153034d27ed218230809f9470d4665"
server_key_version: "encryptionkey@0"
I20260928 10:47:52.664017 24663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 10:47:52.668033 24678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:52.669860 24663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:47:52.670051 24663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "17cc5096295841d0ac39346423fff035"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "d37ed5c5e80e3d4b1df1eb0843a01cf9"
server_key_iv: "c8153034d27ed218230809f9470d4665"
server_key_version: "encryptionkey@0"
I20260928 10:47:52.670248 24663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:52.690521 24663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:52.695516 24663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:52.695792 24663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:52.696822 24663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:52.698351 24663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:52.698431 24663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:52.698529 24663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:52.698572 24663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:52.731285 24663 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:44077
I20260928 10:47:52.731369 24790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:44077 every 8 connection(s)
I20260928 10:47:52.732622 24663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:47:52.737695 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 24663
I20260928 10:47:52.737895 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:47:52.738535 9067 external_mini_cluster.cc:1428] Setting key f954ffefc224176137dbc122698a36d3
Sep 28 10:47:52 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592472, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:52.755136 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:52.735369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:50799 (local address 127.8.218.254:45853)
0928 10:47:52.735843 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:52.735849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:52.736977 (+ 1128us) server_negotiation.cc:407] Connection header received
0928 10:47:52.739051 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:52.739057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:52.739148 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:52.739245 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:52.740938 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:52.741725 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:52.743522 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:52.743957 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:52.744077 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:52.748586 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:52.748592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:52.748595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:52.748648 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:52.751337 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:52.752372 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:52.752377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:52.752380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:52.752515 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:52.753050 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:52.753055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:52.753057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:52.753381 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:52.753577 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:52.754476 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 10:47:52.754763 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":215,"threads_started":1}
I20260928 10:47:52.756310 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:52.735958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:50799)
0928 10:47:52.736748 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:52.736841 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:52.738710 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:52.739543 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:52.739551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:52.740127 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:52.740747 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:52.740766 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:52.741899 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:52.741903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:52.743368 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:52.743384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:52.744211 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:52.744219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:52.744481 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:52.745240 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:52.745291 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:52.748228 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:52.751527 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:52.751539 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:52.751560 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:52.752222 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:52.752620 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:52.752624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:52.752628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:52.752894 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:52.753541 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:52.753550 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:52.754027 (+ 477us) client_negotiation.cc:769] Sending connection context
0928 10:47:52.754354 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 10:47:52.754915 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":304,"threads_started":1}
I20260928 10:47:52.758030 24791 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45853
I20260928 10:47:52.758383 24791 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:52.759229 24791 heartbeater.cc:511] Master 127.8.218.254:45853 requested a full tablet report, sending...
I20260928 10:47:52.761482 24602 ts_manager.cc:194] Registered new tserver with Master: 17cc5096295841d0ac39346423fff035 (127.8.218.193:44077)
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:47:52.763172 24602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:50799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:47:52.793861 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45853
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:53.144582 24797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:53.144901 24797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:53.144996 24797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:53.152556 24797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:53.152701 24797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:53.152822 24797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:47:53.163122 24797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:53.164412 24797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:53.165827 24797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:53.180117 24803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:53.180137 24805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:53.180162 24802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:53.184199 24797 server_base.cc:1034] running on GCE node
I20260928 10:47:53.184988 24797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:53.186163 24797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:53.187408 24797 hybrid_clock.cc:648] HybridClock initialized: now 1790592473187339 us; error 66 us; skew 500 ppm
I20260928 10:47:53.187783 24797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:53 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592473, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:53.191674 24797 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:47:53.193464 24797 webserver.cc:460] Webserver started at http://127.8.218.194:40187/ using document root <none> and password file <none>
I20260928 10:47:53.194243 24797 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:53.194370 24797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:53.194686 24797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:53.197991 24797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "1c61c7d0bf934a629f0b0b1c8304f129"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "2c823cb65bd3d82918a7f79d212b1b9d"
server_key_iv: "808d468feb080819404bcd13874cbb14"
server_key_version: "encryptionkey@0"
I20260928 10:47:53.198837 24797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "1c61c7d0bf934a629f0b0b1c8304f129"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "2c823cb65bd3d82918a7f79d212b1b9d"
server_key_iv: "808d468feb080819404bcd13874cbb14"
server_key_version: "encryptionkey@0"
I20260928 10:47:53.205236 24797 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:47:53.209389 24812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:53.211392 24797 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:47:53.211628 24797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "1c61c7d0bf934a629f0b0b1c8304f129"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "2c823cb65bd3d82918a7f79d212b1b9d"
server_key_iv: "808d468feb080819404bcd13874cbb14"
server_key_version: "encryptionkey@0"
I20260928 10:47:53.212093 24797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:53.228994 24797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:53.233994 24797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:53.234330 24797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:53.235471 24797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:53.237162 24797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:53.237250 24797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:53.237372 24797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:53.237427 24797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:53.270236 24797 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:36743
I20260928 10:47:53.270303 24924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:36743 every 8 connection(s)
I20260928 10:47:53.271987 24797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:47:53.278024 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 24797
I20260928 10:47:53.278249 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:47:53.278806 9067 external_mini_cluster.cc:1428] Setting key 06a8169c71f9f203328dddb70b0131b7
Sep 28 10:47:53 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592473, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:53.297232 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.275229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:41245 (local address 127.8.218.254:45853)
0928 10:47:53.275692 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:53.275697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:53.276898 (+ 1201us) server_negotiation.cc:407] Connection header received
0928 10:47:53.279476 (+ 2578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:53.279484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:53.279589 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:53.279725 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:53.281984 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.282773 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.285184 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.285794 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.285915 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.290618 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:53.290626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:53.290629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:53.290690 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:53.293638 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:53.294559 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:53.294563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:53.294566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:53.294722 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:53.295234 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:53.295239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:53.295241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:53.295532 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:53.295683 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:53.296464 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 10:47:53.296737 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260928 10:47:53.299316 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.275835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:41245)
0928 10:47:53.276712 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:53.276758 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:53.279059 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:53.280109 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:53.280119 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:53.280729 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:53.281646 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.281764 (+ 118us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.282958 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.282965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.284885 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.284905 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.286058 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.286065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.286387 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.287209 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:53.287237 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:53.290152 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:53.293851 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:53.293860 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:53.293878 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:53.294400 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:53.294854 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:53.294860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:53.294862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:53.295099 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:53.295669 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:53.295678 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:53.296068 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 10:47:53.296336 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 10:47:53.296919 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":320,"threads_started":1}
I20260928 10:47:53.301115 24925 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45853
I20260928 10:47:53.301474 24925 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:47:53.302335 24925 heartbeater.cc:511] Master 127.8.218.254:45853 requested a full tablet report, sending...
I20260928 10:47:53.304163 24602 ts_manager.cc:194] Registered new tserver with Master: 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194:36743)
I20260928 10:47:53.305122 24602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:41245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:47:53.339453 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45853
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:47:53.690043 24931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:47:53.690356 24931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:47:53.690500 24931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:47:53.698426 24931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:47:53.698585 24931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:47:53.698742 24931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:47:53.708927 24931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:47:53.710321 24931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:47:53.711666 24931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:47:53.725533 24937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:53.725574 24936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:47:53.725816 24939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:47:53.726740 24931 server_base.cc:1034] running on GCE node
I20260928 10:47:53.727448 24931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:47:53.728502 24931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:47:53.729718 24931 hybrid_clock.cc:648] HybridClock initialized: now 1790592473729673 us; error 27 us; skew 500 ppm
I20260928 10:47:53.730059 24931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:47:53 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592473, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:47:53.734160 24931 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:47:53.736043 24931 webserver.cc:460] Webserver started at http://127.8.218.195:40573/ using document root <none> and password file <none>
I20260928 10:47:53.736801 24931 fs_manager.cc:359] Metadata directory not provided
I20260928 10:47:53.736925 24931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:47:53.737257 24931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:47:53.740343 24931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "37f4322c9eb74c778a52aa222c26aec9"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "7d292970dbb347649e34b0681672073a"
server_key_iv: "4d57e7daef6050fcaaf8a493bf831d9b"
server_key_version: "encryptionkey@0"
I20260928 10:47:53.741209 24931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "37f4322c9eb74c778a52aa222c26aec9"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "7d292970dbb347649e34b0681672073a"
server_key_iv: "4d57e7daef6050fcaaf8a493bf831d9b"
server_key_version: "encryptionkey@0"
I20260928 10:47:53.748070 24931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260928 10:47:53.752286 24946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:53.753979 24931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:47:53.754184 24931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "37f4322c9eb74c778a52aa222c26aec9"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "7d292970dbb347649e34b0681672073a"
server_key_iv: "4d57e7daef6050fcaaf8a493bf831d9b"
server_key_version: "encryptionkey@0"
I20260928 10:47:53.754396 24931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:47:53.767314 24791 heartbeater.cc:503] Master 127.8.218.254:45853 was elected leader, sending a full tablet report...
I20260928 10:47:53.773619 24931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:47:53.777725 24931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:47:53.777964 24931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:47:53.778873 24931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:47:53.780186 24931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:47:53.780277 24931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:53.780372 24931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:47:53.780416 24931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:47:53.812448 24931 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:34053
I20260928 10:47:53.812487 25058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:34053 every 8 connection(s)
I20260928 10:47:53.813822 24931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:47:53.823698 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 24931
I20260928 10:47:53.823886 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:47:53.824348 9067 external_mini_cluster.cc:1428] Setting key 5703035af1996d4eb41e9a423c582d10
Sep 28 10:47:53 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592473, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:47:53.837469 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.816530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:38199 (local address 127.8.218.254:45853)
0928 10:47:53.816966 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:53.816971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:53.818019 (+ 1048us) server_negotiation.cc:407] Connection header received
0928 10:47:53.820036 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:53.820043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:53.820147 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:53.820247 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:53.822120 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.822912 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.824869 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.825519 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.825660 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.831045 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:53.831052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:53.831055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:53.831096 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:53.833956 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:53.834864 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:53.834869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:53.834872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:53.835007 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:53.835472 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:53.835477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:53.835479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:53.835770 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:53.835925 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:53.836755 (+ 830us) server_negotiation.cc:299] Negotiation successful
0928 10:47:53.837027 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":237,"threads_started":1}
I20260928 10:47:53.838502 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.817061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:38199)
0928 10:47:53.817843 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:53.817892 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:53.819711 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:53.820521 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:53.820532 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:53.821105 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:47:53.821888 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.821915 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.823141 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.823145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.824707 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.824723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.825798 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.825808 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.826129 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.827829 (+ 1700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:47:53.827869 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:47:53.830750 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:47:53.834160 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:53.834171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:53.834190 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:53.834702 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:53.835125 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:47:53.835130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:47:53.835134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:47:53.835344 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:47:53.835934 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:47:53.835944 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:47:53.836354 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 10:47:53.836618 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 10:47:53.837214 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":329,"threads_started":1}
I20260928 10:47:53.840154 25059 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45853
I20260928 10:47:53.840521 25059 heartbeater.cc:464] Registering TS with master...
I20260928 10:47:53.841408 25059 heartbeater.cc:511] Master 127.8.218.254:45853 requested a full tablet report, sending...
I20260928 10:47:53.843240 24601 ts_manager.cc:194] Registered new tserver with Master: 37f4322c9eb74c778a52aa222c26aec9 (127.8.218.195:34053)
I20260928 10:47:53.844192 24601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:38199
I20260928 10:47:53.854338 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:47:53.872669 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.859194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.8.218.254:45853)
0928 10:47:53.859438 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:53.859445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:53.859742 (+ 297us) server_negotiation.cc:407] Connection header received
0928 10:47:53.859958 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:53.859962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:53.860030 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:53.860125 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:53.861758 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.862502 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.863876 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.864308 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.864437 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.866880 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:53.866886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:53.866889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:53.866929 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:53.869606 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:53.870406 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:53.870411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:53.870414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:53.870550 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:53.870966 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:53.870972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:53.870975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:53.871420 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:53.871690 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:53.872064 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 10:47:53.872327 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:47:53.878414 24601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:47:53.886639 24601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:47:53.914271 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.900679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.8.218.193:44077)
0928 10:47:53.905264 (+ 4585us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:53.905275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:53.906086 (+ 811us) server_negotiation.cc:407] Connection header received
0928 10:47:53.906213 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:53.906222 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:53.906520 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:53.906719 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:53.907419 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.909087 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.911560 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.913044 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.913188 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.913464 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:53.913747 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 10:47:53.913925 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4389,"thread_start_us":4222,"threads_started":1}
I20260928 10:47:53.914536 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.901813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:36743 (local address 127.0.0.1:59108)
0928 10:47:53.906635 (+ 4822us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:53.906655 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:53.906814 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:53.907708 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:53.907713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:53.907737 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:53.908157 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.908170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.909789 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.909794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.911606 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.911620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.913619 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.913625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.913806 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.913839 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 10:47:53.913925 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 10:47:53.914077 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4683,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":4524,"threads_started":1}
I20260928 10:47:53.914661 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.902310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.8.218.194:36743)
0928 10:47:53.906173 (+ 3863us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:53.906185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:53.906946 (+ 761us) server_negotiation.cc:407] Connection header received
0928 10:47:53.907056 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:53.907064 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:53.907299 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:53.907492 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:53.908375 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.909670 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.911793 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.913173 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.913306 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.913522 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:53.914207 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 10:47:53.914376 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3682,"thread_start_us":3558,"threads_started":1}
I20260928 10:47:53.915331 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.900137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:34616)
0928 10:47:53.905765 (+ 5628us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:53.905787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:53.905932 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:53.906747 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:53.906755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:53.906816 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:53.907199 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.907212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.909234 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.909239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.911394 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.911407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.913236 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.913242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.913474 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.913531 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 10:47:53.913671 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 10:47:53.913882 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5494,"spinlock_wait_cycles":20992,"thread_start_us":2573,"threads_started":1}
I20260928 10:47:53.915433 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.900009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:34053 (local address 127.0.0.1:35350)
0928 10:47:53.902153 (+ 2144us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:53.902222 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:53.902404 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:53.908318 (+ 5914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:53.908322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:53.908349 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:53.908672 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.908681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.910759 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.910765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.912838 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:53.912854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.914656 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.914663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:53.914841 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.914873 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:47:53.914932 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 10:47:53.915067 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"thread_start_us":5442,"threads_started":1}
I20260928 10:47:53.916350 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:53.900199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.8.218.195:34053)
0928 10:47:53.907693 (+ 7494us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:53.907701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:53.907729 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:47:53.907818 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:53.907824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:53.908034 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:53.908815 (+ 781us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:53.909030 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.910611 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.913009 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:53.914410 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:53.915643 (+ 1233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:53.915893 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:53.915961 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 10:47:53.916111 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7316,"thread_start_us":6915,"threads_started":1}
I20260928 10:47:53.918933 24860 tablet_service.cc:1467] Processing CreateTablet for tablet dcfaa9dcbf324892ba3d2311808f0636 (DEFAULT_TABLE table=test-table [id=b812a41abd1a468bb5758553f3d8002f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:53.918978 24726 tablet_service.cc:1467] Processing CreateTablet for tablet dcfaa9dcbf324892ba3d2311808f0636 (DEFAULT_TABLE table=test-table [id=b812a41abd1a468bb5758553f3d8002f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:53.920468 24726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcfaa9dcbf324892ba3d2311808f0636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:53.920567 24860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcfaa9dcbf324892ba3d2311808f0636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:53.920097 24994 tablet_service.cc:1467] Processing CreateTablet for tablet dcfaa9dcbf324892ba3d2311808f0636 (DEFAULT_TABLE table=test-table [id=b812a41abd1a468bb5758553f3d8002f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:47:53.921638 24994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcfaa9dcbf324892ba3d2311808f0636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:47:53.933220 25078 tablet_bootstrap.cc:492] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Bootstrap starting.
I20260928 10:47:53.935701 25079 tablet_bootstrap.cc:492] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Bootstrap starting.
I20260928 10:47:53.937150 25078 tablet_bootstrap.cc:654] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:53.937305 25080 tablet_bootstrap.cc:492] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Bootstrap starting.
I20260928 10:47:53.938745 25078 log.cc:824] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:53.939481 25079 tablet_bootstrap.cc:654] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:53.940570 25080 tablet_bootstrap.cc:654] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Neither blocks nor log segments found. Creating new log.
I20260928 10:47:53.940843 25079 log.cc:824] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:53.941919 25080 log.cc:824] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Log is configured to *not* fsync() on all Append() calls
I20260928 10:47:53.942482 25078 tablet_bootstrap.cc:492] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: No bootstrap required, opened a new log
I20260928 10:47:53.942808 25078 ts_tablet_manager.cc:1397] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260928 10:47:53.944620 25079 tablet_bootstrap.cc:492] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: No bootstrap required, opened a new log
I20260928 10:47:53.944905 25079 ts_tablet_manager.cc:1397] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260928 10:47:53.952500 25080 tablet_bootstrap.cc:492] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: No bootstrap required, opened a new log
I20260928 10:47:53.952803 25080 ts_tablet_manager.cc:1397] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260928 10:47:53.953668 25078 raft_consensus.cc:348] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:53.954155 25078 raft_consensus.cc:374] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:53.954252 25078 raft_consensus.cc:729] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cc5096295841d0ac39346423fff035, State: Initialized, Role: FOLLOWER
I20260928 10:47:53.954913 25078 consensus_queue.cc:260] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:53.955483 25079 raft_consensus.cc:348] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:53.956013 25079 raft_consensus.cc:374] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:53.956115 25079 raft_consensus.cc:729] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c61c7d0bf934a629f0b0b1c8304f129, State: Initialized, Role: FOLLOWER
I20260928 10:47:53.956729 25079 consensus_queue.cc:260] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:53.957935 24925 heartbeater.cc:503] Master 127.8.218.254:45853 was elected leader, sending a full tablet report...
I20260928 10:47:53.960019 25079 ts_tablet_manager.cc:1428] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260928 10:47:53.961750 25078 ts_tablet_manager.cc:1428] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Time spent starting tablet: real 0.019s user 0.004s sys 0.011s
I20260928 10:47:53.961687 25080 raft_consensus.cc:348] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:53.962109 25080 raft_consensus.cc:374] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:47:53.962217 25080 raft_consensus.cc:729] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f4322c9eb74c778a52aa222c26aec9, State: Initialized, Role: FOLLOWER
I20260928 10:47:53.963001 25080 consensus_queue.cc:260] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:53.964164 25059 heartbeater.cc:503] Master 127.8.218.254:45853 was elected leader, sending a full tablet report...
I20260928 10:47:53.964834 25080 ts_tablet_manager.cc:1428] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260928 10:47:53.985158 24792 tablet.cc:2369] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:47:54.024278 24926 tablet.cc:2369] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:54.032384 25085 raft_consensus.cc:482] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:47:54.032737 25085 raft_consensus.cc:504] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:54.034598 25085 leader_election.cc:288] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37f4322c9eb74c778a52aa222c26aec9 (127.8.218.195:34053), 17cc5096295841d0ac39346423fff035 (127.8.218.193:44077)
I20260928 10:47:54.035633 25086 raft_consensus.cc:482] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:47:54.035964 25086 raft_consensus.cc:504] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:54.038921 25086 leader_election.cc:288] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17cc5096295841d0ac39346423fff035 (127.8.218.193:44077), 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194:36743)
I20260928 10:47:54.044013 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.035143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:39791 (local address 127.8.218.193:44077)
0928 10:47:54.035346 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:54.035352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:54.036385 (+ 1033us) server_negotiation.cc:407] Connection header received
0928 10:47:54.036664 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:54.036669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:54.036744 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:54.036826 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:54.037554 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.038567 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.041831 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.042981 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.043140 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.043218 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:54.043596 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 10:47:54.043739 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:47:54.044443 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.035007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.8.218.194:39791)
0928 10:47:54.036274 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:54.036294 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:54.036494 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:54.036939 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:54.036942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:54.036962 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:54.037403 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:54.037415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.039187 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.039194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:54.041628 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:54.041648 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.043191 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.043207 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:54.043359 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.043455 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 10:47:54.043542 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 10:47:54.043783 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":17,"thread_start_us":523,"threads_started":1}
I20260928 10:47:54.045023 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.035191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:56971 (local address 127.8.218.195:34053)
0928 10:47:54.035490 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:54.035496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:54.036130 (+ 634us) server_negotiation.cc:407] Connection header received
0928 10:47:54.036501 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:54.036507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:54.036591 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:54.036688 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:54.037420 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.038589 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.042298 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.043820 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.043981 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.044052 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:54.044625 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 10:47:54.044772 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260928 10:47:54.045094 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.034953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:34053 (local address 127.8.218.194:56971)
0928 10:47:54.035997 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:54.036019 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:54.036277 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:54.036706 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:54.036710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:54.036733 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:54.037198 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:54.037212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.039821 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.039826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:54.042100 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:54.042117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.044243 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.044249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:54.044463 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.044496 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 10:47:54.044592 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 10:47:54.044730 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":20,"thread_start_us":1033,"threads_started":1}
I20260928 10:47:54.045940 24746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfaa9dcbf324892ba3d2311808f0636" candidate_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cc5096295841d0ac39346423fff035" is_pre_election: true
I20260928 10:47:54.046651 24746 raft_consensus.cc:2454] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c61c7d0bf934a629f0b0b1c8304f129 in term 0.
I20260928 10:47:54.047400 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.038217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.8.218.195:59753)
0928 10:47:54.038634 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:54.038652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:54.038758 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:54.040344 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:54.040349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:54.040378 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:54.040642 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:54.040659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.042455 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.042460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:54.044491 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:54.044505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.046560 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.046570 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:54.046801 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.046835 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:47:54.046939 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 10:47:54.047090 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260928 10:47:54.047513 24816 leader_election.cc:302] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cc5096295841d0ac39346423fff035, 1c61c7d0bf934a629f0b0b1c8304f129; no voters:
I20260928 10:47:54.047580 25014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfaa9dcbf324892ba3d2311808f0636" candidate_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f4322c9eb74c778a52aa222c26aec9" is_pre_election: true
I20260928 10:47:54.048054 25085 raft_consensus.cc:2781] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:47:54.048055 25014 raft_consensus.cc:2454] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c61c7d0bf934a629f0b0b1c8304f129 in term 0.
I20260928 10:47:54.048210 25085 raft_consensus.cc:482] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:47:54.048321 25085 raft_consensus.cc:3037] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:54.048826 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.038816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:59753 (local address 127.8.218.193:44077)
0928 10:47:54.040029 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:54.040035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:54.040056 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:47:54.040116 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:54.040120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:54.040192 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:54.040798 (+ 606us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:54.041073 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.042320 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.045308 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.046404 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.046590 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.046673 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:54.048421 (+ 1748us) server_negotiation.cc:299] Negotiation successful
0928 10:47:54.048587 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":826,"threads_started":1}
I20260928 10:47:54.049853 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.038632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:47167 (local address 127.8.218.194:36743)
0928 10:47:54.038818 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:54.038823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:54.039701 (+ 878us) server_negotiation.cc:407] Connection header received
0928 10:47:54.039978 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:54.039984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:54.040066 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:54.040125 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:47:54.040955 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.042116 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.046828 (+ 4712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.048106 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.048263 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.048338 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:54.049405 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0928 10:47:54.049594 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:47:54.049860 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.038244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:36743 (local address 127.8.218.195:47167)
0928 10:47:54.039612 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0928 10:47:54.039630 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:47:54.039818 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:47:54.040313 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:47:54.040318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:47:54.040344 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:47:54.040776 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:54.040787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.042367 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.042372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:54.046644 (+ 4272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:47:54.046662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.048999 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.049006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:47:54.049187 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.049222 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:47:54.049321 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 10:47:54.049491 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"mutex_wait_us":3,"thread_start_us":1043,"threads_started":1}
I20260928 10:47:54.050706 24746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfaa9dcbf324892ba3d2311808f0636" candidate_uuid: "37f4322c9eb74c778a52aa222c26aec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cc5096295841d0ac39346423fff035" is_pre_election: true
I20260928 10:47:54.050961 24746 raft_consensus.cc:2454] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f4322c9eb74c778a52aa222c26aec9 in term 0.
I20260928 10:47:54.050870 24880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfaa9dcbf324892ba3d2311808f0636" candidate_uuid: "37f4322c9eb74c778a52aa222c26aec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" is_pre_election: true
I20260928 10:47:54.051518 24950 leader_election.cc:302] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cc5096295841d0ac39346423fff035, 37f4322c9eb74c778a52aa222c26aec9; no voters:
I20260928 10:47:54.051654 25085 raft_consensus.cc:504] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:54.051995 25086 raft_consensus.cc:2781] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:47:54.052081 24880 raft_consensus.cc:2379] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37f4322c9eb74c778a52aa222c26aec9 in current term 1: Already voted for candidate 1c61c7d0bf934a629f0b0b1c8304f129 in this term.
I20260928 10:47:54.052163 25086 raft_consensus.cc:482] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:47:54.052271 25086 raft_consensus.cc:3037] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:54.053552 25014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfaa9dcbf324892ba3d2311808f0636" candidate_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f4322c9eb74c778a52aa222c26aec9"
I20260928 10:47:54.053545 24746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfaa9dcbf324892ba3d2311808f0636" candidate_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cc5096295841d0ac39346423fff035"
I20260928 10:47:54.053776 24746 raft_consensus.cc:3037] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:47:54.055328 25086 raft_consensus.cc:504] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:54.055858 25014 raft_consensus.cc:2379] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c61c7d0bf934a629f0b0b1c8304f129 in current term 1: Already voted for candidate 37f4322c9eb74c778a52aa222c26aec9 in this term.
I20260928 10:47:54.056792 24746 raft_consensus.cc:2454] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c61c7d0bf934a629f0b0b1c8304f129 in term 1.
I20260928 10:47:54.057097 25085 leader_election.cc:288] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [CANDIDATE]: Term 1 election: Requested vote from peers 37f4322c9eb74c778a52aa222c26aec9 (127.8.218.195:34053), 17cc5096295841d0ac39346423fff035 (127.8.218.193:44077)
I20260928 10:47:54.057289 24816 leader_election.cc:302] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cc5096295841d0ac39346423fff035, 1c61c7d0bf934a629f0b0b1c8304f129; no voters:
I20260928 10:47:54.057677 25085 raft_consensus.cc:2781] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:47:54.058164 25085 raft_consensus.cc:686] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 LEADER]: Becoming Leader. State: Replica: 1c61c7d0bf934a629f0b0b1c8304f129, State: Running, Role: LEADER
I20260928 10:47:54.058718 25086 leader_election.cc:288] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [CANDIDATE]: Term 1 election: Requested vote from peers 17cc5096295841d0ac39346423fff035 (127.8.218.193:44077), 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194:36743)
I20260928 10:47:54.058694 24746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfaa9dcbf324892ba3d2311808f0636" candidate_uuid: "37f4322c9eb74c778a52aa222c26aec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cc5096295841d0ac39346423fff035"
I20260928 10:47:54.058831 25085 consensus_queue.cc:237] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:47:54.059221 24746 raft_consensus.cc:2379] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 37f4322c9eb74c778a52aa222c26aec9 in current term 1: Already voted for candidate 1c61c7d0bf934a629f0b0b1c8304f129 in this term.
I20260928 10:47:54.059314 24880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfaa9dcbf324892ba3d2311808f0636" candidate_uuid: "37f4322c9eb74c778a52aa222c26aec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c61c7d0bf934a629f0b0b1c8304f129"
I20260928 10:47:54.060079 24950 leader_election.cc:302] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37f4322c9eb74c778a52aa222c26aec9; no voters: 17cc5096295841d0ac39346423fff035, 1c61c7d0bf934a629f0b0b1c8304f129
I20260928 10:47:54.060379 25086 raft_consensus.cc:2726] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 10:47:54.064994 24601 catalog_manager.cc:5495] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194). New cstate: current_term: 1 leader_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } health_report { overall_health: HEALTHY } } }
W20260928 10:47:54.065495 25060 tablet.cc:2369] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:47:54.107100 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.101385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.8.218.194:36743)
0928 10:47:54.101656 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:54.101662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:54.101684 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:47:54.101827 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:54.101832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:54.101917 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:54.102057 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:47:54.102452 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.103207 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.104772 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.105206 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.105323 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.105921 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:47:54.106465 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:47:54.106605 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:54.106719 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 10:47:54.106811 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260928 10:47:54.115562 24746 raft_consensus.cc:1261] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 1 FOLLOWER]: Refusing update from remote peer 1c61c7d0bf934a629f0b0b1c8304f129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:54.115613 25014 raft_consensus.cc:1261] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Refusing update from remote peer 1c61c7d0bf934a629f0b0b1c8304f129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:47:54.116474 25091 consensus_queue.cc:1035] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:54.116883 25085 consensus_queue.cc:1035] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:47:54.135510 25095 mvcc.cc:205] Tried to move back new op lower bound from 7334266773967667200 to 7334266773750472704. Current Snapshot: MvccSnapshot[applied={T|T < 7334266773967667200}]
I20260928 10:47:54.149834 24602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:47:54.150158 24602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:47:54.154275 24602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Sending DeleteTablet for 3 replicas of tablet dcfaa9dcbf324892ba3d2311808f0636
I20260928 10:47:54.155671 24726 tablet_service.cc:1514] Processing DeleteTablet for tablet dcfaa9dcbf324892ba3d2311808f0636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:54 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:34616
I20260928 10:47:54.155673 24994 tablet_service.cc:1514] Processing DeleteTablet for tablet dcfaa9dcbf324892ba3d2311808f0636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:54 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:35350
I20260928 10:47:54.155678 24860 tablet_service.cc:1514] Processing DeleteTablet for tablet dcfaa9dcbf324892ba3d2311808f0636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:47:54 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:59108
I20260928 10:47:54.158003 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 24663
I20260928 10:47:54.166775 25106 tablet_replica.cc:330] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: stopping tablet replica
I20260928 10:47:54.167450 25106 raft_consensus.cc:2229] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:47:54.168262 25106 raft_consensus.cc:2258] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:54.169303 25105 tablet_replica.cc:330] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: stopping tablet replica
I20260928 10:47:54.169835 25105 raft_consensus.cc:2229] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:47:54.170423 25105 raft_consensus.cc:2258] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:54.173182 25104 tablet_replica.cc:330] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: stopping tablet replica
I20260928 10:47:54.173743 25104 raft_consensus.cc:2229] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:47:54.174340 25104 raft_consensus.cc:2258] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:47:54.174921 25106 ts_tablet_manager.cc:1905] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:54.176851 25104 ts_tablet_manager.cc:1905] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:54.177126 25105 ts_tablet_manager.cc:1905] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:47:54.184378 25104 ts_tablet_manager.cc:1918] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:54.184593 25104 log.cc:1196] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/dcfaa9dcbf324892ba3d2311808f0636
I20260928 10:47:54.185158 25104 ts_tablet_manager.cc:1939] T dcfaa9dcbf324892ba3d2311808f0636 P 17cc5096295841d0ac39346423fff035: Deleting consensus metadata
Sep 28 10:47:54 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:47:54.187564 24589 catalog_manager.cc:4841] TS 17cc5096295841d0ac39346423fff035 (127.8.218.193:44077): tablet dcfaa9dcbf324892ba3d2311808f0636 (table test-table [id=b812a41abd1a468bb5758553f3d8002f]) successfully deleted
I20260928 10:47:54.190405 25105 ts_tablet_manager.cc:1918] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:54.190629 25105 log.cc:1196] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/dcfaa9dcbf324892ba3d2311808f0636
I20260928 10:47:54.191337 25105 ts_tablet_manager.cc:1939] T dcfaa9dcbf324892ba3d2311808f0636 P 37f4322c9eb74c778a52aa222c26aec9: Deleting consensus metadata
I20260928 10:47:54.191622 25106 ts_tablet_manager.cc:1918] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:47:54.191869 25106 log.cc:1196] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/dcfaa9dcbf324892ba3d2311808f0636
I20260928 10:47:54.192523 25106 ts_tablet_manager.cc:1939] T dcfaa9dcbf324892ba3d2311808f0636 P 1c61c7d0bf934a629f0b0b1c8304f129: Deleting consensus metadata
I20260928 10:47:54.193578 24587 catalog_manager.cc:4841] TS 37f4322c9eb74c778a52aa222c26aec9 (127.8.218.195:34053): tablet dcfaa9dcbf324892ba3d2311808f0636 (table test-table [id=b812a41abd1a468bb5758553f3d8002f]) successfully deleted
I20260928 10:47:54.194420 24586 catalog_manager.cc:4841] TS 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194:36743): tablet dcfaa9dcbf324892ba3d2311808f0636 (table test-table [id=b812a41abd1a468bb5758553f3d8002f]) successfully deleted
I20260928 10:47:54.195820 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.159597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.8.218.193:44077)
0928 10:47:54.159809 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:54.159816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:54.174076 (+ 14260us) server_negotiation.cc:407] Connection header received
0928 10:47:54.174415 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:54.174422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:54.174511 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:54.174593 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:54.176732 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.177902 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.179974 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.180662 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.180773 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.187008 (+ 6235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:54.187026 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:54.187044 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:54.187104 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:54.191542 (+ 4438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:54.192953 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:54.192965 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:54.192969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:54.193171 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:54.193765 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:54.193770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:54.193773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:54.194344 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:54.194676 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:54.195052 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 10:47:54.195336 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:47:54.196643 24786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:54.342768 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 24663
I20260928 10:47:54.357965 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 24797
Sep 28 10:47:54 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:47:54.377686 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.358762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.8.218.194:36743)
0928 10:47:54.358947 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:54.358973 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:54.358996 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:47:54.359230 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:54.359236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:54.359324 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:54.359423 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:54.361159 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.361969 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.363922 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.364452 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.364562 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.370437 (+ 5875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:54.370458 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:54.370479 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:54.370536 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:54.374184 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:54.375247 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:54.375257 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:54.375259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:54.375431 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:54.375905 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:54.375911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:54.375915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:54.376446 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:54.376708 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:54.377093 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 10:47:54.377341 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:47:54.378558 24920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:54.495930 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 24797
I20260928 10:47:54.511627 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 24931
Sep 28 10:47:54 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:47:54.528873 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:47:54.512434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.8.218.195:34053)
0928 10:47:54.512589 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 10:47:54.512593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:47:54.512623 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:47:54.512708 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:47:54.512712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:47:54.512792 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:47:54.512888 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:47:54.514453 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.515429 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.517028 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:47:54.517629 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:47:54.517785 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:47:54.522733 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:47:54.522751 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:47:54.522768 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:47:54.522818 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:47:54.525689 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:54.526591 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:54.526600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:54.526603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:54.526770 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:47:54.527230 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:47:54.527234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:47:54.527237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:47:54.527631 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:47:54.527875 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:47:54.528312 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 10:47:54.528496 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 10:47:54.529605 25054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:54.621971 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 24931
I20260928 10:47:54.636477 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 24570
I20260928 10:47:54.637735 24632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:47:54.754019 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 24570
I20260928 10:48:14.774926 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45853
--webserver_interface=127.8.218.254
--webserver_port=41433
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:48:15.119591 25112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:15.119910 25112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:15.120004 25112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:15.127521 25112 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 10:48:15.127655 25112 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 10:48:15.127693 25112 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 10:48:15.127748 25112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:48:15.127805 25112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:15.127874 25112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:15.127912 25112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:48:15.127967 25112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:48:15.138424 25112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=41433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:15.139866 25112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:15.141314 25112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:15.165855 25121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:15.165822 25119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:15.165858 25112 server_base.cc:1034] running on GCE node
W20260928 10:48:15.165905 25118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:15.167212 25112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:15.168977 25112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:15.170356 25112 hybrid_clock.cc:648] HybridClock initialized: now 1790592495170263 us; error 61 us; skew 500 ppm
I20260928 10:48:15.170827 25112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:15 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592495, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:15.174825 25112 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:48:15.176716 25112 webserver.cc:460] Webserver started at http://127.8.218.254:41433/ using document root <none> and password file <none>
I20260928 10:48:15.177618 25112 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:15.177742 25112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:15.185348 25112 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 10:48:15.189816 25128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:15.191442 25112 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:48:15.191642 25112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "01e389a5846a4df693105dfc43533618"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "79a46c7f273a8193cfe6bf951539ae75"
server_key_iv: "2d7b46b188e9bb1ec0afa3a7d310c19b"
server_key_version: "encryptionkey@0"
I20260928 10:48:15.192757 25112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:15.215576 25112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:15.221024 25112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:15.221354 25112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:15.240572 25112 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:45853
I20260928 10:48:15.240619 25179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:45853 every 8 connection(s)
I20260928 10:48:15.242225 25112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:48:15.249945 25180 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:48:15.251327 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 25112
I20260928 10:48:15.252162 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:44077
--local_ip_for_outbound_sockets=127.8.218.193
--tserver_master_addrs=127.8.218.254:45853
--webserver_port=44109
--webserver_interface=127.8.218.193
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:48:15.253626 25180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Bootstrap starting.
I20260928 10:48:15.274381 25180 log.cc:824] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:15.287227 25180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:48:15.287806 25180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Bootstrap complete.
I20260928 10:48:15.298755 25180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:48:15.299754 25180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e389a5846a4df693105dfc43533618, State: Initialized, Role: FOLLOWER
I20260928 10:48:15.300441 25180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:48:15.300714 25180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:48:15.300841 25180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:48:15.301023 25180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 1 FOLLOWER]: Advancing to term 2
I20260928 10:48:15.306597 25180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:48:15.307271 25180 leader_election.cc:302] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e389a5846a4df693105dfc43533618; no voters:
I20260928 10:48:15.307964 25180 leader_election.cc:288] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 10:48:15.308174 25184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 10:48:15.309661 25184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 2 LEADER]: Becoming Leader. State: Replica: 01e389a5846a4df693105dfc43533618, State: Running, Role: LEADER
I20260928 10:48:15.310336 25184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:48:15.311048 25180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:48:15.313652 25185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "01e389a5846a4df693105dfc43533618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } } }
I20260928 10:48:15.314342 25185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: This master's current role is: LEADER
I20260928 10:48:15.315224 25186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e389a5846a4df693105dfc43533618. Latest consensus state: current_term: 2 leader_uuid: "01e389a5846a4df693105dfc43533618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } } }
I20260928 10:48:15.315634 25186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: This master's current role is: LEADER
I20260928 10:48:15.323374 25189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:48:15.333225 25189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:48:15.335750 25189 catalog_manager.cc:1259] Loaded cluster ID: 981eb65b3c5443ca8b56586bff8238f3
I20260928 10:48:15.336047 25189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:48:15.340979 25189 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:48:15.343468 25189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Loaded TSK: 0
I20260928 10:48:15.378300 25189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Generated new TSK 1
I20260928 10:48:15.378886 25189 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:48:15.639307 25182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:15.639765 25182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:15.639968 25182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:15.650876 25182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:15.651017 25182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:15.651187 25182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:48:15.663595 25182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:44077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=44109
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:15.664966 25182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:15.666407 25182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:15.681093 25208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:15.681627 25207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:15.681777 25210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:15.682405 25182 server_base.cc:1034] running on GCE node
I20260928 10:48:15.683207 25182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:15.684330 25182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:15.685540 25182 hybrid_clock.cc:648] HybridClock initialized: now 1790592495685472 us; error 58 us; skew 500 ppm
I20260928 10:48:15.685899 25182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:15 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592495, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:15.689316 25182 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:48:15.691171 25182 webserver.cc:460] Webserver started at http://127.8.218.193:44109/ using document root <none> and password file <none>
I20260928 10:48:15.691905 25182 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:15.692044 25182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:15.698021 25182 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:48:15.701040 25217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:15.702696 25182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:48:15.702890 25182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "17cc5096295841d0ac39346423fff035"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "d37ed5c5e80e3d4b1df1eb0843a01cf9"
server_key_iv: "c8153034d27ed218230809f9470d4665"
server_key_version: "encryptionkey@0"
I20260928 10:48:15.703962 25182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:15.728144 25182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:15.732790 25182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:15.733063 25182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:15.734181 25182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:15.735647 25182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:15.735755 25182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:15.735862 25182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:15.735929 25182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:15.766768 25182 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:44077
I20260928 10:48:15.766834 25329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:44077 every 8 connection(s)
I20260928 10:48:15.768219 25182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:48:15.777842 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 25182
I20260928 10:48:15.778426 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:36743
--local_ip_for_outbound_sockets=127.8.218.194
--tserver_master_addrs=127.8.218.254:45853
--webserver_port=40187
--webserver_interface=127.8.218.194
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:48:15 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592495, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:48:15.792217 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:15.771105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:33737 (local address 127.8.218.254:45853)
0928 10:48:15.771721 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:15.771739 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:15.772331 (+ 592us) server_negotiation.cc:407] Connection header received
0928 10:48:15.774463 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:15.774476 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:15.774845 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:15.775177 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:15.776793 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:15.778164 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:15.779813 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:15.780453 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:15.780642 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:15.784345 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:15.784354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:15.784371 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:15.784421 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:15.787492 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:15.788414 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:15.788422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:15.788427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:15.788599 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:15.789107 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:15.789111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:15.789113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:15.789712 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:15.789888 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:15.790888 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0928 10:48:15.791247 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":279,"threads_started":1}
I20260928 10:48:15.792423 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:15.771366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:33737)
0928 10:48:15.772152 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:15.772194 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:15.773945 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:15.775381 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:15.775393 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:15.776027 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:48:15.776584 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:15.776602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:15.778350 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:15.778354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:15.779670 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:15.779683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:15.780693 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:15.780700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:15.780953 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:15.781630 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:48:15.781654 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:48:15.784042 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:48:15.787661 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:15.787671 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:15.787687 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:15.788268 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:15.788699 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:15.788704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:15.788708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:15.788968 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:15.789864 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:48:15.789872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:48:15.790298 (+ 426us) client_negotiation.cc:769] Sending connection context
0928 10:48:15.790608 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 10:48:15.791187 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":329,"threads_started":1}
I20260928 10:48:15.795506 25330 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45853
I20260928 10:48:15.795795 25330 heartbeater.cc:464] Registering TS with master...
I20260928 10:48:15.796563 25330 heartbeater.cc:511] Master 127.8.218.254:45853 requested a full tablet report, sending...
I20260928 10:48:15.798913 25145 ts_manager.cc:194] Registered new tserver with Master: 17cc5096295841d0ac39346423fff035 (127.8.218.193:44077)
I20260928 10:48:15.800491 25145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:33737
W20260928 10:48:16.112278 25334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:16.112583 25334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:16.112679 25334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:16.120185 25334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:16.120329 25334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:16.120448 25334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:48:16.130682 25334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:36743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=40187
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:16.132031 25334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:16.133289 25334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:16.146716 25339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:16.146683 25342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:16.146756 25340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:16.151273 25334 server_base.cc:1034] running on GCE node
I20260928 10:48:16.152046 25334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:16.153065 25334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:16.154279 25334 hybrid_clock.cc:648] HybridClock initialized: now 1790592496154211 us; error 66 us; skew 500 ppm
I20260928 10:48:16.154609 25334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:16 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592496, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:16.158108 25334 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:48:16.159981 25334 webserver.cc:460] Webserver started at http://127.8.218.194:40187/ using document root <none> and password file <none>
I20260928 10:48:16.160734 25334 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:16.160888 25334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:16.166706 25334 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:48:16.170047 25349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:16.171824 25334 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260928 10:48:16.172052 25334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "1c61c7d0bf934a629f0b0b1c8304f129"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "2c823cb65bd3d82918a7f79d212b1b9d"
server_key_iv: "808d468feb080819404bcd13874cbb14"
server_key_version: "encryptionkey@0"
I20260928 10:48:16.173156 25334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:16.185124 25334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:16.190099 25334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:16.190379 25334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:16.191423 25334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:16.192819 25334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:16.192931 25334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:16.193054 25334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:16.193123 25334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:16.226415 25334 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:36743
I20260928 10:48:16.226500 25461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:36743 every 8 connection(s)
I20260928 10:48:16.227887 25334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:48:16.233395 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 25334
I20260928 10:48:16.234225 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:34053
--local_ip_for_outbound_sockets=127.8.218.195
--tserver_master_addrs=127.8.218.254:45853
--webserver_port=40573
--webserver_interface=127.8.218.195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 10:48:16 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592496, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:48:16.252403 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.230777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:54705 (local address 127.8.218.254:45853)
0928 10:48:16.230999 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.231003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.232219 (+ 1216us) server_negotiation.cc:407] Connection header received
0928 10:48:16.234707 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.234714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.234818 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.234917 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:16.237518 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.238286 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.240913 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.241693 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.241858 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.246222 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:16.246227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:16.246231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:16.246279 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:16.248939 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:16.249843 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:16.249848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:16.249850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:16.249987 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:16.250461 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:16.250465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:16.250467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:16.250803 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:16.250948 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.251756 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.252038 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:48:16.253563 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.231262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:54705)
0928 10:48:16.232040 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:16.232085 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:16.234326 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:16.235321 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:16.235335 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:16.236180 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:48:16.237108 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.237141 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.238530 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.238537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.240662 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.240688 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.241976 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.241984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.242308 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.243191 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:48:16.243226 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:48:16.245935 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:48:16.249113 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:16.249124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:16.249143 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:16.249689 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:16.250109 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:16.250114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:16.250118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:16.250332 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:16.250955 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:48:16.250963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:48:16.251353 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 10:48:16.251617 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 10:48:16.252194 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":285,"threads_started":1}
I20260928 10:48:16.255210 25462 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45853
I20260928 10:48:16.255599 25462 heartbeater.cc:464] Registering TS with master...
I20260928 10:48:16.256429 25462 heartbeater.cc:511] Master 127.8.218.254:45853 requested a full tablet report, sending...
I20260928 10:48:16.258030 25145 ts_manager.cc:194] Registered new tserver with Master: 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194:36743)
I20260928 10:48:16.258876 25145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:54705
W20260928 10:48:16.582432 25465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:16.582722 25465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:16.582854 25465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:16.590205 25465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:16.590330 25465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:16.590493 25465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:48:16.600577 25465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=40573
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:16.601893 25465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:16.603175 25465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:16.616276 25470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:16.616269 25471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:16.616422 25473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:16.617143 25465 server_base.cc:1034] running on GCE node
I20260928 10:48:16.617843 25465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:16.618845 25465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:16.620074 25465 hybrid_clock.cc:648] HybridClock initialized: now 1790592496620017 us; error 44 us; skew 500 ppm
I20260928 10:48:16.620409 25465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:16 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592496, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:16.623805 25465 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:48:16.625479 25465 webserver.cc:460] Webserver started at http://127.8.218.195:40573/ using document root <none> and password file <none>
I20260928 10:48:16.626199 25465 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:16.626340 25465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:16.631994 25465 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 10:48:16.635121 25480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:16.636618 25465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:48:16.636793 25465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "37f4322c9eb74c778a52aa222c26aec9"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "7d292970dbb347649e34b0681672073a"
server_key_iv: "4d57e7daef6050fcaaf8a493bf831d9b"
server_key_version: "encryptionkey@0"
I20260928 10:48:16.637842 25465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:16.656615 25465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:16.660930 25465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:16.661152 25465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:16.662253 25465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:16.663563 25465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:16.663651 25465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:16.663734 25465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:16.663786 25465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:16.695150 25465 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:34053
I20260928 10:48:16.695187 25592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:34053 every 8 connection(s)
I20260928 10:48:16.696700 25465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:48:16.705730 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 25465
Sep 28 10:48:16 dist-test-slave-mn38 krb5kdc[24557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592496, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:48:16.718436 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.699368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:33489 (local address 127.8.218.254:45853)
0928 10:48:16.699540 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.699544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.700686 (+ 1142us) server_negotiation.cc:407] Connection header received
0928 10:48:16.702542 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.702546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.702617 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.702704 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:16.704209 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.704966 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.707231 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.707894 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.708001 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.712293 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:16.712300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:16.712304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:16.712353 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:16.715147 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:16.715977 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:16.715984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:16.715987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:16.716118 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:16.716557 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:16.716561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:16.716563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:16.716868 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:16.717013 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.717789 (+ 776us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.718039 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 10:48:16.719516 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.699761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:33489)
0928 10:48:16.700520 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:16.700562 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:16.702294 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:16.702988 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:16.702997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:16.703485 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:48:16.704045 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.704063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.705119 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.705123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.706999 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.707015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.708161 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.708168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.708434 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.709108 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:48:16.709132 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:48:16.712044 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:48:16.715330 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:16.715342 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:16.715361 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:16.715839 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:16.716236 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:16.716241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:16.716245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:16.716442 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:16.717001 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:48:16.717009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:48:16.717375 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 10:48:16.717672 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 10:48:16.718273 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":284,"threads_started":1}
I20260928 10:48:16.721217 25593 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45853
I20260928 10:48:16.721586 25593 heartbeater.cc:464] Registering TS with master...
I20260928 10:48:16.721647 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.706681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.8.218.254:45853)
0928 10:48:16.707194 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.707200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.707525 (+ 325us) server_negotiation.cc:407] Connection header received
0928 10:48:16.707699 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.707703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.707767 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.707862 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:16.709496 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.710577 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.712277 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.712782 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.712895 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.715866 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:16.715872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:16.715875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:16.715911 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:16.718477 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:16.719399 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:16.719406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:16.719409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:16.719539 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:16.720003 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:16.720009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:16.720012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:16.720414 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:16.720701 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.721008 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.721248 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":254,"threads_started":1}
I20260928 10:48:16.722565 25593 heartbeater.cc:511] Master 127.8.218.254:45853 requested a full tablet report, sending...
I20260928 10:48:16.724370 25144 ts_manager.cc:194] Registered new tserver with Master: 37f4322c9eb74c778a52aa222c26aec9 (127.8.218.195:34053)
I20260928 10:48:16.725339 25144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:33489
I20260928 10:48:16.737337 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:48:16.744328 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.738518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.8.218.254:45853)
0928 10:48:16.738776 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.738782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.739054 (+ 272us) server_negotiation.cc:407] Connection header received
0928 10:48:16.739312 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.739316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.739474 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.739615 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:48:16.740131 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.740850 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.742366 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.742860 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.742976 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.743461 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:48:16.743664 (+ 203us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 10:48:16.744041 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41436: token expired
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 10:48:16.744441 25596 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41436: token expired
W20260928 10:48:16.744938 25064 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.218.254:45853)
I20260928 10:48:16.758018 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.745462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.8.218.254:45853)
0928 10:48:16.745623 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.745628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.745645 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:48:16.745732 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.745737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.745812 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.745893 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:16.747245 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.747963 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.749332 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.749786 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.749899 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.752634 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:16.752639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:16.752643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:16.752681 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:16.755162 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:16.755917 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:16.755923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:16.755927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:16.756068 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:16.756457 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:16.756462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:16.756466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:16.756821 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:16.757050 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.757422 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.757689 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 10:48:16.761842 25144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:48:16.769158 25144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:48:16.803735 25330 heartbeater.cc:503] Master 127.8.218.254:45853 was elected leader, sending a full tablet report...
I20260928 10:48:16.807016 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.782160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.8.218.193:44077)
0928 10:48:16.796913 (+ 14753us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.796924 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.796962 (+ 38us) server_negotiation.cc:407] Connection header received
0928 10:48:16.797140 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.797150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.797508 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.798643 (+ 1135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:16.798898 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.800423 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.803736 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.805274 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.805426 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.805692 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.806492 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.806672 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14574,"thread_start_us":14407,"threads_started":1}
I20260928 10:48:16.807081 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.782148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.8.218.194:36743)
0928 10:48:16.799045 (+ 16897us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.799053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.799082 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:48:16.799171 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.799179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.799440 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.799635 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:16.800241 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.801653 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.804116 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.805498 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.805617 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.805803 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.806596 (+ 793us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.806776 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16728,"thread_start_us":16712,"threads_started":1}
I20260928 10:48:16.807070 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.781532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:41944)
0928 10:48:16.795939 (+ 14407us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:16.795962 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:16.796110 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:16.797797 (+ 1687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:16.797804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:16.797843 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:16.798411 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.798424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.800567 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.800572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.803517 (+ 2945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.803540 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.805866 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.805874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.806154 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.806282 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 10:48:16.806418 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 10:48:16.806648 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14251,"mutex_wait_us":65,"thread_start_us":16131,"threads_started":1}
I20260928 10:48:16.807989 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.781677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:36743 (local address 127.0.0.1:55802)
0928 10:48:16.798010 (+ 16333us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:16.798034 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:16.798169 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:16.799668 (+ 1499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:16.799681 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:16.799721 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:16.800060 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.800071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.801829 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.801835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.803923 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.803936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.805955 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.805961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.806171 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.806275 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 10:48:16.806418 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 10:48:16.807588 (+ 1170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16166,"mutex_wait_us":1013,"thread_start_us":14525,"threads_started":1}
I20260928 10:48:16.808701 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.781295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:34053 (local address 127.0.0.1:50318)
0928 10:48:16.793581 (+ 12286us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:16.793651 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:16.793912 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:16.800818 (+ 6906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:16.800823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:16.800847 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:16.801666 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.801678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.803348 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.803354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.805951 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.805964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.807779 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.807786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.807968 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.808006 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:48:16.808091 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 10:48:16.808222 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11917,"mutex_wait_us":4,"thread_start_us":12843,"threads_started":1}
I20260928 10:48:16.809491 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.781536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.8.218.195:34053)
0928 10:48:16.800293 (+ 18757us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.800301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.800329 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:48:16.800421 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.800429 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.800637 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.800804 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:16.801981 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.803217 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.806133 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.807555 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.807726 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.807984 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.808193 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.808357 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18589,"thread_start_us":18448,"threads_started":1}
I20260928 10:48:16.811905 25265 tablet_service.cc:1467] Processing CreateTablet for tablet 33865a8a6a564dbdba735cbcc0149e10 (DEFAULT_TABLE table=test-table [id=fd94ef55423b4407bb29e7ef268ef7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:48:16.811904 25397 tablet_service.cc:1467] Processing CreateTablet for tablet 33865a8a6a564dbdba735cbcc0149e10 (DEFAULT_TABLE table=test-table [id=fd94ef55423b4407bb29e7ef268ef7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:48:16.812950 25528 tablet_service.cc:1467] Processing CreateTablet for tablet 33865a8a6a564dbdba735cbcc0149e10 (DEFAULT_TABLE table=test-table [id=fd94ef55423b4407bb29e7ef268ef7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:48:16.813503 25397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33865a8a6a564dbdba735cbcc0149e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:48:16.813536 25265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33865a8a6a564dbdba735cbcc0149e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:48:16.814313 25528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33865a8a6a564dbdba735cbcc0149e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:48:16.827646 25609 tablet_bootstrap.cc:492] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Bootstrap starting.
I20260928 10:48:16.830195 25610 tablet_bootstrap.cc:492] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: Bootstrap starting.
I20260928 10:48:16.833055 25609 tablet_bootstrap.cc:654] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Neither blocks nor log segments found. Creating new log.
I20260928 10:48:16.834045 25611 tablet_bootstrap.cc:492] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: Bootstrap starting.
I20260928 10:48:16.834088 25610 tablet_bootstrap.cc:654] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: Neither blocks nor log segments found. Creating new log.
I20260928 10:48:16.834481 25609 log.cc:824] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:16.835413 25610 log.cc:824] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:16.837690 25611 tablet_bootstrap.cc:654] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: Neither blocks nor log segments found. Creating new log.
I20260928 10:48:16.838124 25609 tablet_bootstrap.cc:492] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: No bootstrap required, opened a new log
I20260928 10:48:16.838392 25609 ts_tablet_manager.cc:1397] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260928 10:48:16.838923 25611 log.cc:824] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:16.840412 25610 tablet_bootstrap.cc:492] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: No bootstrap required, opened a new log
I20260928 10:48:16.840719 25610 ts_tablet_manager.cc:1397] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 10:48:16.841579 25611 tablet_bootstrap.cc:492] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: No bootstrap required, opened a new log
I20260928 10:48:16.841872 25611 ts_tablet_manager.cc:1397] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260928 10:48:16.849200 25609 raft_consensus.cc:348] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.849514 25611 raft_consensus.cc:348] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.849817 25609 raft_consensus.cc:374] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:48:16.849917 25611 raft_consensus.cc:374] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:48:16.849961 25609 raft_consensus.cc:729] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f4322c9eb74c778a52aa222c26aec9, State: Initialized, Role: FOLLOWER
I20260928 10:48:16.850019 25611 raft_consensus.cc:729] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cc5096295841d0ac39346423fff035, State: Initialized, Role: FOLLOWER
I20260928 10:48:16.850637 25611 consensus_queue.cc:260] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.850751 25609 consensus_queue.cc:260] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.851622 25610 raft_consensus.cc:348] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.852332 25610 raft_consensus.cc:374] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:48:16.852478 25610 raft_consensus.cc:729] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c61c7d0bf934a629f0b0b1c8304f129, State: Initialized, Role: FOLLOWER
I20260928 10:48:16.852514 25611 ts_tablet_manager.cc:1428] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260928 10:48:16.853493 25593 heartbeater.cc:503] Master 127.8.218.254:45853 was elected leader, sending a full tablet report...
I20260928 10:48:16.853649 25610 consensus_queue.cc:260] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.854254 25609 ts_tablet_manager.cc:1428] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260928 10:48:16.855155 25462 heartbeater.cc:503] Master 127.8.218.254:45853 was elected leader, sending a full tablet report...
I20260928 10:48:16.856096 25610 ts_tablet_manager.cc:1428] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 10:48:16.891054 25615 raft_consensus.cc:482] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:48:16.891348 25615 raft_consensus.cc:504] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.893204 25615 leader_election.cc:288] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37f4322c9eb74c778a52aa222c26aec9 (127.8.218.195:34053), 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194:36743)
I20260928 10:48:16.900445 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.893803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:52533 (local address 127.8.218.194:36743)
0928 10:48:16.894044 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.894051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.894701 (+ 650us) server_negotiation.cc:407] Connection header received
0928 10:48:16.894889 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.894893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.894953 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.895027 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:16.895665 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.896376 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.898616 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.899411 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.899560 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.899636 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.900034 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.900192 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 10:48:16.900432 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.893616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:57617 (local address 127.8.218.195:34053)
0928 10:48:16.893887 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.893895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.894582 (+ 687us) server_negotiation.cc:407] Connection header received
0928 10:48:16.894814 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.894818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.894881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.894957 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:16.895633 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.896370 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.898550 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.899359 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.899521 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.899603 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.899993 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.900131 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:48:16.900638 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.893524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:34053 (local address 127.8.218.193:57617)
0928 10:48:16.894452 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:16.894471 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:16.894643 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:16.894989 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:16.894993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:16.895018 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:16.895478 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.895493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.896510 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.896514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.898378 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.898394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.899591 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.899599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.899758 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.899823 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 10:48:16.899908 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 10:48:16.900052 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":141,"spinlock_wait_cycles":26368,"thread_start_us":547,"threads_started":1}
I20260928 10:48:16.900852 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.893692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:36743 (local address 127.8.218.193:52533)
0928 10:48:16.894582 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:16.894596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:16.894758 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:16.895057 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:16.895062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:16.895091 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:16.895507 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.895516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.896506 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.896511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.898455 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:16.898468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.899626 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.899632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:16.899847 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.899873 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 10:48:16.899954 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 10:48:16.900086 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":130,"thread_start_us":573,"threads_started":1}
I20260928 10:48:16.901794 25548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33865a8a6a564dbdba735cbcc0149e10" candidate_uuid: "17cc5096295841d0ac39346423fff035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f4322c9eb74c778a52aa222c26aec9" is_pre_election: true
I20260928 10:48:16.901801 25417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33865a8a6a564dbdba735cbcc0149e10" candidate_uuid: "17cc5096295841d0ac39346423fff035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" is_pre_election: true
I20260928 10:48:16.902256 25417 raft_consensus.cc:2454] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17cc5096295841d0ac39346423fff035 in term 0.
I20260928 10:48:16.902253 25548 raft_consensus.cc:2454] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17cc5096295841d0ac39346423fff035 in term 0.
I20260928 10:48:16.902892 25218 leader_election.cc:302] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cc5096295841d0ac39346423fff035, 1c61c7d0bf934a629f0b0b1c8304f129; no voters:
I20260928 10:48:16.903258 25615 raft_consensus.cc:2781] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:48:16.903411 25615 raft_consensus.cc:482] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:48:16.903509 25615 raft_consensus.cc:3037] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:48:16.905778 25615 raft_consensus.cc:504] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.906487 25615 leader_election.cc:288] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [CANDIDATE]: Term 1 election: Requested vote from peers 37f4322c9eb74c778a52aa222c26aec9 (127.8.218.195:34053), 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194:36743)
I20260928 10:48:16.906919 25548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33865a8a6a564dbdba735cbcc0149e10" candidate_uuid: "17cc5096295841d0ac39346423fff035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f4322c9eb74c778a52aa222c26aec9"
I20260928 10:48:16.907001 25417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33865a8a6a564dbdba735cbcc0149e10" candidate_uuid: "17cc5096295841d0ac39346423fff035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c61c7d0bf934a629f0b0b1c8304f129"
I20260928 10:48:16.907143 25548 raft_consensus.cc:3037] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:48:16.907193 25417 raft_consensus.cc:3037] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:48:16.910557 25548 raft_consensus.cc:2454] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17cc5096295841d0ac39346423fff035 in term 1.
I20260928 10:48:16.910557 25417 raft_consensus.cc:2454] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17cc5096295841d0ac39346423fff035 in term 1.
I20260928 10:48:16.911060 25218 leader_election.cc:302] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cc5096295841d0ac39346423fff035, 1c61c7d0bf934a629f0b0b1c8304f129; no voters:
I20260928 10:48:16.911384 25615 raft_consensus.cc:2781] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:48:16.911967 25615 raft_consensus.cc:686] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 1 LEADER]: Becoming Leader. State: Replica: 17cc5096295841d0ac39346423fff035, State: Running, Role: LEADER
I20260928 10:48:16.912462 25615 consensus_queue.cc:237] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } }
I20260928 10:48:16.917132 25144 catalog_manager.cc:5495] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17cc5096295841d0ac39346423fff035 (127.8.218.193). New cstate: current_term: 1 leader_uuid: "17cc5096295841d0ac39346423fff035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17cc5096295841d0ac39346423fff035" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 } health_report { overall_health: UNKNOWN } } }
I20260928 10:48:16.945187 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.939289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.8.218.193:44077)
0928 10:48:16.939474 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.939478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.939497 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:48:16.939673 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.939676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.939832 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.939971 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:48:16.940393 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.941197 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.942828 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.943261 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:16.943372 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:16.943969 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:48:16.944491 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:48:16.944611 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:16.944844 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 10:48:16.944919 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 10:48:16.948289 25594 tablet.cc:2369] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:48:16.953944 25417 raft_consensus.cc:1261] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 FOLLOWER]: Refusing update from remote peer 17cc5096295841d0ac39346423fff035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:48:16.953943 25548 raft_consensus.cc:1261] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Refusing update from remote peer 17cc5096295841d0ac39346423fff035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:48:16.955080 25620 consensus_queue.cc:1035] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37f4322c9eb74c778a52aa222c26aec9" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:48:16.955478 25615 consensus_queue.cc:1035] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c61c7d0bf934a629f0b0b1c8304f129" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 36743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:48:16.970228 25623 mvcc.cc:205] Tried to move back new op lower bound from 7334266867512045568 to 7334266867357724672. Current Snapshot: MvccSnapshot[applied={T|T < 7334266867512045568}]
I20260928 10:48:16.979813 25625 mvcc.cc:205] Tried to move back new op lower bound from 7334266867512045568 to 7334266867357724672. Current Snapshot: MvccSnapshot[applied={T|T < 7334266867512045568}]
I20260928 10:48:16.985178 25144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:48:16.985508 25144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:48:16.989496 25144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Sending DeleteTablet for 3 replicas of tablet 33865a8a6a564dbdba735cbcc0149e10
I20260928 10:48:16.990754 25528 tablet_service.cc:1514] Processing DeleteTablet for tablet 33865a8a6a564dbdba735cbcc0149e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:48:16 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:50318
I20260928 10:48:16.990798 25265 tablet_service.cc:1514] Processing DeleteTablet for tablet 33865a8a6a564dbdba735cbcc0149e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:48:16 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:41944
I20260928 10:48:16.990957 25397 tablet_service.cc:1514] Processing DeleteTablet for tablet 33865a8a6a564dbdba735cbcc0149e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:48:16 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:55802
I20260928 10:48:16.992218 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 25182
I20260928 10:48:17.001818 25632 tablet_replica.cc:330] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: stopping tablet replica
I20260928 10:48:17.003207 25632 raft_consensus.cc:2229] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:48:17.004765 25632 raft_consensus.cc:2258] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:48:17.005604 25633 tablet_replica.cc:330] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: stopping tablet replica
I20260928 10:48:17.006244 25633 raft_consensus.cc:2229] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:48:17.007198 25633 raft_consensus.cc:2258] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:48:17.007684 25634 tablet_replica.cc:330] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: stopping tablet replica
I20260928 10:48:17.008172 25634 raft_consensus.cc:2229] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:48:17.008780 25634 raft_consensus.cc:2258] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:48:17.010265 25632 ts_tablet_manager.cc:1905] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:48:17.011036 25633 ts_tablet_manager.cc:1905] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:48:17.012560 25634 ts_tablet_manager.cc:1905] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:48:17.018949 25633 ts_tablet_manager.cc:1918] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:48:17.019244 25633 log.cc:1196] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/33865a8a6a564dbdba735cbcc0149e10
I20260928 10:48:17.019856 25633 ts_tablet_manager.cc:1939] T 33865a8a6a564dbdba735cbcc0149e10 P 37f4322c9eb74c778a52aa222c26aec9: Deleting consensus metadata
I20260928 10:48:17.020032 25634 ts_tablet_manager.cc:1918] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:48:17.020200 25634 log.cc:1196] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/33865a8a6a564dbdba735cbcc0149e10
I20260928 10:48:17.020704 25634 ts_tablet_manager.cc:1939] T 33865a8a6a564dbdba735cbcc0149e10 P 1c61c7d0bf934a629f0b0b1c8304f129: Deleting consensus metadata
I20260928 10:48:17.022037 25130 catalog_manager.cc:4841] TS 37f4322c9eb74c778a52aa222c26aec9 (127.8.218.195:34053): tablet 33865a8a6a564dbdba735cbcc0149e10 (table test-table [id=fd94ef55423b4407bb29e7ef268ef7cc]) successfully deleted
I20260928 10:48:17.022469 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:16.993934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.8.218.193:44077)
0928 10:48:16.994111 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:16.994117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:16.994136 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:48:16.994195 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:16.994200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:16.994281 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:16.994383 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:16.996473 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:16.997760 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:17.000000 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:17.000706 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:17.000839 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:17.004878 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:17.004899 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:17.004918 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:17.004975 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:17.010507 (+ 5532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:17.012035 (+ 1528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:17.012046 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:17.012050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:17.012241 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:17.019892 (+ 7651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:17.019899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:17.019903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:17.021071 (+ 1168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:17.021521 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:17.021848 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 10:48:17.022114 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:48:17.022817 25129 catalog_manager.cc:4841] TS 1c61c7d0bf934a629f0b0b1c8304f129 (127.8.218.194:36743): tablet 33865a8a6a564dbdba735cbcc0149e10 (table test-table [id=fd94ef55423b4407bb29e7ef268ef7cc]) successfully deleted
I20260928 10:48:17.021582 25632 ts_tablet_manager.cc:1918] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:48:17.023133 25632 log.cc:1196] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/33865a8a6a564dbdba735cbcc0149e10
I20260928 10:48:17.023332 25325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:17.119882 25632 ts_tablet_manager.cc:1939] T 33865a8a6a564dbdba735cbcc0149e10 P 17cc5096295841d0ac39346423fff035: Deleting consensus metadata
I20260928 10:48:17.120570 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 25182
W20260928 10:48:17.135777 25132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.218.193:44077 (error 108)
I20260928 10:48:17.135921 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 25334
I20260928 10:48:17.137902 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.137362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:41972)
0928 10:48:17.137583 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:17.137717 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 10:48:17.138159 25132 catalog_manager.cc:4584] TS 17cc5096295841d0ac39346423fff035 (127.8.218.193:44077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33865a8a6a564dbdba735cbcc0149e10: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44077: connect: Connection refused (error 111)
I20260928 10:48:17.155194 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.136904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.8.218.194:36743)
0928 10:48:17.137063 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:17.137068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:17.137085 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:48:17.137229 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:17.137234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:17.137320 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:17.137423 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:17.139081 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:17.139911 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:17.141843 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:17.142302 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:17.142411 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:17.148569 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:17.148588 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:17.148605 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:17.148684 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:17.151822 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:17.152796 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:17.152806 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:17.152808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:17.152997 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:17.153522 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:17.153527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:17.153529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:17.154023 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:17.154277 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:17.154670 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 10:48:17.154920 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:48:17.155995 25457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:17.191874 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.191333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:41982)
0928 10:48:17.191583 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:17.191700 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 10:48:17.192876 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.192544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:41998)
0928 10:48:17.192660 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:17.192760 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 10:48:17.246923 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 25334
I20260928 10:48:17.253428 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.252927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:42012)
0928 10:48:17.253145 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:17.253271 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 10:48:17.254561 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.254188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:42018)
0928 10:48:17.254352 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:17.254429 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 10:48:17.268746 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 25465
I20260928 10:48:17.282567 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.269511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.8.218.195:34053)
0928 10:48:17.269669 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:17.269675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:17.269693 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:48:17.269809 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:17.269813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:17.269872 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:17.269966 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:17.271309 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:17.272062 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:17.273518 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:17.273937 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:17.274038 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:17.276845 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:17.276862 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:17.276879 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:17.276918 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:17.279699 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:17.280534 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:17.280542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:17.280544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:17.280715 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:17.281063 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:17.281066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:17.281068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:17.281430 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:17.281772 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:17.282134 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 10:48:17.282319 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 10:48:17.283239 25588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:17.335229 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.334716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:42020)
0928 10:48:17.334966 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:17.335069 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 10:48:17.336284 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:17.335877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44077 (local address 127.0.0.1:42026)
0928 10:48:17.336076 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:17.336162 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 10:48:17.372783 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 25465
I20260928 10:48:17.386340 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 25112
I20260928 10:48:17.387517 25175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:17.503288 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 25112
I20260928 10:48:37.522375 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45853
--webserver_interface=127.8.218.254
--webserver_port=41433
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:48:37.852998 25640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:37.853314 25640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:37.853464 25640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:37.860901 25640 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 10:48:37.861008 25640 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 10:48:37.861047 25640 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 10:48:37.861123 25640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:48:37.861188 25640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:37.861274 25640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:37.861341 25640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:48:37.861405 25640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:48:37.871712 25640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=41433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:37.873003 25640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:37.874356 25640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:37.885838 25645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:37.885891 25646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:37.886348 25648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:37.886538 25640 server_base.cc:1034] running on GCE node
I20260928 10:48:37.887318 25640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:37.888797 25640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:37.890034 25640 hybrid_clock.cc:648] HybridClock initialized: now 1790592517889965 us; error 42 us; skew 500 ppm
I20260928 10:48:37.890367 25640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:37 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592517, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:37.897140 25640 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:48:37.898820 25640 webserver.cc:460] Webserver started at http://127.8.218.254:41433/ using document root <none> and password file <none>
I20260928 10:48:37.899547 25640 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:37.899663 25640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:37.905283 25640 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:48:37.908133 25655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:37.909590 25640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:48:37.909749 25640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "01e389a5846a4df693105dfc43533618"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "79a46c7f273a8193cfe6bf951539ae75"
server_key_iv: "2d7b46b188e9bb1ec0afa3a7d310c19b"
server_key_version: "encryptionkey@0"
I20260928 10:48:37.910789 25640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:37.936156 25640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:37.940269 25640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:37.940537 25640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:37.958284 25640 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:45853
I20260928 10:48:37.958320 25706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:45853 every 8 connection(s)
I20260928 10:48:37.959743 25640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:48:37.966013 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 25640
I20260928 10:48:37.966996 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:44077
--local_ip_for_outbound_sockets=127.8.218.193
--tserver_master_addrs=127.8.218.254:45853
--webserver_port=44109
--webserver_interface=127.8.218.193
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:48:37.967136 25707 sys_catalog.cc:263] Verifying existing consensus state
I20260928 10:48:37.971662 25707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Bootstrap starting.
I20260928 10:48:37.993181 25707 log.cc:824] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:38.009773 25707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 10:48:38.010335 25707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Bootstrap complete.
I20260928 10:48:38.019618 25707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:48:38.020501 25707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e389a5846a4df693105dfc43533618, State: Initialized, Role: FOLLOWER
I20260928 10:48:38.021130 25707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:48:38.021334 25707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:48:38.021481 25707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:48:38.021649 25707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 2 FOLLOWER]: Advancing to term 3
I20260928 10:48:38.025425 25707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:48:38.026044 25707 leader_election.cc:302] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e389a5846a4df693105dfc43533618; no voters:
I20260928 10:48:38.026654 25707 leader_election.cc:288] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 10:48:38.026795 25711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 10:48:38.027830 25711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [term 3 LEADER]: Becoming Leader. State: Replica: 01e389a5846a4df693105dfc43533618, State: Running, Role: LEADER
I20260928 10:48:38.028455 25711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } }
I20260928 10:48:38.028777 25707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:48:38.031025 25713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e389a5846a4df693105dfc43533618. Latest consensus state: current_term: 3 leader_uuid: "01e389a5846a4df693105dfc43533618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } } }
I20260928 10:48:38.031064 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "01e389a5846a4df693105dfc43533618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e389a5846a4df693105dfc43533618" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45853 } } }
I20260928 10:48:38.031396 25713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: This master's current role is: LEADER
I20260928 10:48:38.031426 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618 [sys.catalog]: This master's current role is: LEADER
I20260928 10:48:38.037205 25718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:48:38.058159 25718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:48:38.060165 25718 catalog_manager.cc:1259] Loaded cluster ID: 981eb65b3c5443ca8b56586bff8238f3
I20260928 10:48:38.060436 25718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:48:38.065694 25718 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:48:38.068288 25718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Loaded TSK: 0
I20260928 10:48:38.068637 25718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 01e389a5846a4df693105dfc43533618: Loaded TSK: 1
F20260928 10:48:38.070323 25718 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7ab1d659ad google::LogMessage::Fail() at ??:0
@ 0x7f7ab1d6888a google::LogMessage::SendToLog() at ??:0
@ 0x7f7ab1d654de google::LogMessage::Flush() at ??:0
@ 0x7f7ab1d66a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7ab5ac97d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7ab5ab873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7ac77941c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7ac7793783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7ac781f2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7ac781f0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7ac78695e9 std::function<>::operator()() at ??:0
@ 0x7f7ac7797cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7ac7795c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7ac781d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7ac781d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7ab49b1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7ab49b491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7ab49b4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7ab498feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7ab7c696db start_thread at ??:0
@ 0x7f7ab0bff61f clone at ??:0
W20260928 10:48:38.375381 25709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:38.375694 25709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:38.375797 25709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:38.383297 25709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:38.383427 25709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:38.383544 25709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:48:38.393867 25709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:44077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=44109
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:38.395213 25709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:38.396507 25709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:38.409823 25757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:38.409888 25755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:38.409914 25754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:38.411042 25709 server_base.cc:1034] running on GCE node
I20260928 10:48:38.411672 25709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:38.412621 25709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:38.413851 25709 hybrid_clock.cc:648] HybridClock initialized: now 1790592518413790 us; error 44 us; skew 500 ppm
I20260928 10:48:38.414191 25709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:38 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592518, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:38.417531 25709 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:48:38.419152 25709 webserver.cc:460] Webserver started at http://127.8.218.193:44109/ using document root <none> and password file <none>
I20260928 10:48:38.419848 25709 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:38.419967 25709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:38.425589 25709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:48:38.428352 25764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:38.429824 25709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:48:38.429981 25709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "17cc5096295841d0ac39346423fff035"
format_stamp: "Formatted at 2026-09-28 10:47:52 on dist-test-slave-mn38"
server_key: "d37ed5c5e80e3d4b1df1eb0843a01cf9"
server_key_iv: "c8153034d27ed218230809f9470d4665"
server_key_version: "encryptionkey@0"
I20260928 10:48:38.430962 25709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:38.452723 25709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:38.457805 25709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:38.458141 25709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:38.459326 25709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:38.460772 25709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:38.460871 25709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:38.460984 25709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:38.461054 25709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:38.492872 25709 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:44077
I20260928 10:48:38.492929 25876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:44077 every 8 connection(s)
I20260928 10:48:38.494423 25709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:48:38.499686 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:38.497600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:40981)
0928 10:48:38.498385 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:38.498653 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":307,"threads_started":1}
W20260928 10:48:38.500486 25877 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:45853 (0 consecutive failures): Network error: Failed to ping master at 127.8.218.254:45853: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
I20260928 10:48:38.501420 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:38.501060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:48399)
0928 10:48:38.501230 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:38.501304 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 10:48:38.502403 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:38.501977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:52287)
0928 10:48:38.502172 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:38.502264 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 10:48:38.502668 25877 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 10:48:38.504951 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 25709
I20260928 10:48:38.505601 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:36743
--local_ip_for_outbound_sockets=127.8.218.194
--tserver_master_addrs=127.8.218.254:45853
--webserver_port=40187
--webserver_interface=127.8.218.194
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:48:38.847805 25880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:38.848116 25880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:38.848243 25880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:38.855894 25880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:38.856026 25880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:38.856178 25880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:48:38.866510 25880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:36743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=40187
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:38.867862 25880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:38.869283 25880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:38.883111 25886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:38.883066 25885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:38.883188 25888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:38.883910 25880 server_base.cc:1034] running on GCE node
I20260928 10:48:38.884567 25880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:38.885710 25880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:38.886934 25880 hybrid_clock.cc:648] HybridClock initialized: now 1790592518886867 us; error 65 us; skew 500 ppm
I20260928 10:48:38.887275 25880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:38 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592518, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:38.890513 25880 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:48:38.892030 25880 webserver.cc:460] Webserver started at http://127.8.218.194:40187/ using document root <none> and password file <none>
I20260928 10:48:38.892727 25880 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:38.892835 25880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:38.898707 25880 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 10:48:38.901932 25895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:38.903529 25880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:48:38.903723 25880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "1c61c7d0bf934a629f0b0b1c8304f129"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "2c823cb65bd3d82918a7f79d212b1b9d"
server_key_iv: "808d468feb080819404bcd13874cbb14"
server_key_version: "encryptionkey@0"
I20260928 10:48:38.904879 25880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:38.920724 25880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:38.925242 25880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:38.925506 25880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:38.926507 25880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:38.928130 25880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:38.928236 25880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:38.928331 25880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:38.928402 25880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:38.960062 25880 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:36743
I20260928 10:48:38.960117 26007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:36743 every 8 connection(s)
I20260928 10:48:38.961649 25880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:48:38.966835 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:38.964791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:45691)
0928 10:48:38.965621 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:38.965890 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":303,"threads_started":1}
W20260928 10:48:38.967599 26008 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:45853 (0 consecutive failures): Network error: Failed to ping master at 127.8.218.254:45853: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
I20260928 10:48:38.968554 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:38.968129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:41863)
0928 10:48:38.968338 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:38.968428 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 10:48:38.969482 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:38.969057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:50885)
0928 10:48:38.969215 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:38.969312 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 10:48:38.969713 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 25880
W20260928 10:48:38.969769 26008 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 10:48:38.970294 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:34053
--local_ip_for_outbound_sockets=127.8.218.195
--tserver_master_addrs=127.8.218.254:45853
--webserver_port=40573
--webserver_interface=127.8.218.195
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:48:39.308563 26011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:39.308897 26011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:39.309020 26011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:39.316912 26011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:39.317075 26011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:39.317236 26011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:48:39.327991 26011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=40573
--tserver_master_addrs=127.8.218.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:39.329391 26011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:39.330802 26011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:39.345122 26016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:39.345042 26017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:39.345098 26019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:39.345958 26011 server_base.cc:1034] running on GCE node
I20260928 10:48:39.346642 26011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:39.347752 26011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:39.349020 26011 hybrid_clock.cc:648] HybridClock initialized: now 1790592519348945 us; error 64 us; skew 500 ppm
I20260928 10:48:39.349392 26011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:39 dist-test-slave-mn38 krb5kdc[24557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592519, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:39.352893 26011 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:48:39.354743 26011 webserver.cc:460] Webserver started at http://127.8.218.195:40573/ using document root <none> and password file <none>
I20260928 10:48:39.355542 26011 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:39.355755 26011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:39.362293 26011 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:48:39.365425 26026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:39.367028 26011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:48:39.367218 26011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "37f4322c9eb74c778a52aa222c26aec9"
format_stamp: "Formatted at 2026-09-28 10:47:53 on dist-test-slave-mn38"
server_key: "7d292970dbb347649e34b0681672073a"
server_key_iv: "4d57e7daef6050fcaaf8a493bf831d9b"
server_key_version: "encryptionkey@0"
I20260928 10:48:39.368278 26011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:39.385795 26011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:39.390173 26011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:39.390429 26011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:39.391476 26011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:39.392961 26011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:39.393046 26011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:39.393148 26011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:39.393213 26011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:39.425093 26011 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:34053
I20260928 10:48:39.425180 26138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:34053 every 8 connection(s)
I20260928 10:48:39.426592 26011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:48:39.432026 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:39.429912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:38519)
0928 10:48:39.430837 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:39.431094 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":330,"threads_started":1}
W20260928 10:48:39.432790 26139 heartbeater.cc:644] Failed to heartbeat to 127.8.218.254:45853 (0 consecutive failures): Network error: Failed to ping master at 127.8.218.254:45853: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
I20260928 10:48:39.433486 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 26011
I20260928 10:48:39.433877 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:39.433405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:39927)
0928 10:48:39.433615 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:39.433715 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 10:48:39.435138 26141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:39.434605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:44491)
0928 10:48:39.434842 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:39.434956 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 10:48:39.435570 26139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 10:48:39.504312 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:39.503382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:54377)
0928 10:48:39.503974 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:39.504066 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":339,"threads_started":1}
I20260928 10:48:39.971108 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:39.970328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:41027)
0928 10:48:39.970769 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:39.970877 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":213,"threads_started":1}
I20260928 10:48:40.437134 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:40.436162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:40723)
0928 10:48:40.436706 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:40.436851 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":253,"threads_started":1}
I20260928 10:48:40.506346 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:40.505541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:35539)
0928 10:48:40.506069 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:40.506159 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":272,"threads_started":1}
I20260928 10:48:40.973162 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:40.972286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:60493)
0928 10:48:40.972838 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:40.972941 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":271,"threads_started":1}
I20260928 10:48:41.438992 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:41.438211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:45931)
0928 10:48:41.438720 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:41.438813 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
I20260928 10:48:41.508231 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:41.507408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:51547)
0928 10:48:41.507944 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:41.508040 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":258,"threads_started":1}
I20260928 10:48:41.974857 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:41.974068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:41349)
0928 10:48:41.974584 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:41.974678 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":255,"threads_started":1}
I20260928 10:48:42.440862 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:42.440049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:52481)
0928 10:48:42.440571 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:42.440667 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":271,"threads_started":1}
I20260928 10:48:42.510402 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:42.509535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:60913)
0928 10:48:42.510086 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:42.510186 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":253,"threads_started":1}
I20260928 10:48:42.976842 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:42.975951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:53651)
0928 10:48:42.976477 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:42.976578 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260928 10:48:43.442787 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:43.442049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:50005)
0928 10:48:43.442504 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:43.442590 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":224,"threads_started":1}
I20260928 10:48:43.512252 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:43.511472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:45321)
0928 10:48:43.511977 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:43.512071 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260928 10:48:43.978771 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:43.977946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:34407)
0928 10:48:43.978483 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:43.978577 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260928 10:48:44.444689 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:44.443853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:45225)
0928 10:48:44.444376 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:44.444469 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":259,"threads_started":1}
I20260928 10:48:44.514017 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:44.513240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:54275)
0928 10:48:44.513749 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:44.513849 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":273,"threads_started":1}
I20260928 10:48:44.980410 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:44.979697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:38273)
0928 10:48:44.980147 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:44.980246 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260928 10:48:45.446528 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:45.445785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:42983)
0928 10:48:45.446258 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:45.446358 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":236,"threads_started":1}
I20260928 10:48:45.516033 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:45.515140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:45745)
0928 10:48:45.515707 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:45.515808 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":247,"threads_started":1}
I20260928 10:48:45.982384 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:45.981572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:36347)
0928 10:48:45.982080 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:45.982171 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":208,"threads_started":1}
I20260928 10:48:46.448496 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:46.447624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:43465)
0928 10:48:46.448180 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:46.448273 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":242,"threads_started":1}
I20260928 10:48:46.517984 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:46.517147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:55489)
0928 10:48:46.517711 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:46.517818 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":287,"threads_started":1}
I20260928 10:48:46.984228 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:46.983462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:58309)
0928 10:48:46.983958 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:46.984059 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":263,"threads_started":1}
I20260928 10:48:47.450629 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:47.449868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:44159)
0928 10:48:47.450350 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:47.450441 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
I20260928 10:48:47.520200 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:47.519309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:47879)
0928 10:48:47.519875 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:47.519987 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":291,"threads_started":1}
I20260928 10:48:47.986069 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:47.985244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:60061)
0928 10:48:47.985746 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:47.985837 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
I20260928 10:48:48.452548 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:48.451697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:46325)
0928 10:48:48.452263 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:48.452359 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":264,"threads_started":1}
I20260928 10:48:48.522218 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:48.521420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:34783)
0928 10:48:48.521944 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:48.522049 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":246,"threads_started":1}
I20260928 10:48:48.987934 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:48.987163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:53917)
0928 10:48:48.987679 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:48.987772 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":288,"threads_started":1}
I20260928 10:48:49.454453 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:49.453634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:35157)
0928 10:48:49.454180 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:49.454274 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":338,"threads_started":1}
I20260928 10:48:49.524195 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:49.523341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:47743)
0928 10:48:49.523891 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:49.524008 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":236,"threads_started":1}
I20260928 10:48:49.989993 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:49.989104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:39543)
0928 10:48:49.989681 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:49.989804 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":253,"threads_started":1}
I20260928 10:48:50.456655 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:50.455576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:47319)
0928 10:48:50.456306 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:50.456455 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":371,"threads_started":1}
I20260928 10:48:50.526126 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:50.525222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:44051)
0928 10:48:50.525749 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:50.525931 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
I20260928 10:48:50.992305 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:50.991242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:51971)
0928 10:48:50.991815 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:50.992031 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":303,"threads_started":1}
I20260928 10:48:51.458441 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:51.457700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:57751)
0928 10:48:51.458183 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:51.458272 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260928 10:48:51.527841 26179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:51.527078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:60989)
0928 10:48:51.527552 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:51.527651 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":224,"threads_started":1}
I20260928 10:48:51.994221 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:51.993389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:50005)
0928 10:48:51.993916 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:51.994028 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":259,"threads_started":1}
I20260928 10:48:52.460285 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:52.459391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:35869)
0928 10:48:52.459982 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:52.460091 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":277,"threads_started":1}
I20260928 10:48:52.529515 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:52.528739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:57839)
0928 10:48:52.529185 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:52.529291 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260928 10:48:52.995988 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:52.995207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:42663)
0928 10:48:52.995685 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:52.995800 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260928 10:48:53.462278 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:53.461338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:48391)
0928 10:48:53.461943 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:53.462040 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":281,"threads_started":1}
I20260928 10:48:53.531469 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:53.530638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:41599)
0928 10:48:53.531167 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:53.531285 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":286,"threads_started":1}
I20260928 10:48:53.998003 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:53.997161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.194:35223)
0928 10:48:53.997653 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:53.997815 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":268,"threads_started":1}
I20260928 10:48:54.442212 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 25709
I20260928 10:48:54.462270 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:54.443492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.8.218.193:44077)
0928 10:48:54.444124 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:54.444144 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:54.444190 (+ 46us) server_negotiation.cc:407] Connection header received
0928 10:48:54.446098 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:54.446111 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:54.446607 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:54.446967 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:54.448506 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:54.449840 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:54.451183 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:54.451935 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:54.452130 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:54.455158 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:54.455168 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:54.455191 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:54.455249 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:54.458475 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:54.459387 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:54.459396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:54.459405 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:54.459630 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:54.460078 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:54.460082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:54.460084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:54.460709 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:54.460973 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:54.461402 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 10:48:54.461969 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":274,"threads_started":1}
I20260928 10:48:54.463989 25872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:54.464366 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:54.463554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.195:36863)
0928 10:48:54.464062 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:54.464177 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":273,"threads_started":1}
I20260928 10:48:54.550663 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 25709
I20260928 10:48:54.550745 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:54.549584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45853 (local address 127.8.218.193:59355)
0928 10:48:54.550213 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:54.550413 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.254:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":347,"threads_started":1}
I20260928 10:48:54.566442 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 25880
I20260928 10:48:54.586400 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:54.567354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.8.218.194:36743)
0928 10:48:54.567860 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:54.567874 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:54.567911 (+ 37us) server_negotiation.cc:407] Connection header received
0928 10:48:54.569553 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:54.569565 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:54.570108 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:54.570418 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:54.571908 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:54.573496 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:54.575003 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:54.575799 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:54.575960 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:54.578734 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:54.578745 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:54.578765 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:54.578830 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:54.582384 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:54.583211 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:54.583220 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:54.583228 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:54.583432 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:54.583913 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:54.583918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:54.583920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:54.584598 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:54.584974 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:54.585576 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 10:48:54.586079 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":226,"threads_started":1}
I20260928 10:48:54.587975 26003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:54.674208 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 25880
I20260928 10:48:54.688385 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 26011
I20260928 10:48:54.708858 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:54.689344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.8.218.195:34053)
0928 10:48:54.689988 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:54.690003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:54.690047 (+ 44us) server_negotiation.cc:407] Connection header received
0928 10:48:54.691806 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:54.691818 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:54.692327 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:54.692694 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:54.694315 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:54.695646 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:54.697071 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:54.697937 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:54.698151 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:54.701217 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:54.701227 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:54.701247 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:54.701312 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:54.705040 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:54.705962 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:54.705971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:54.705979 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:54.706190 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:54.706632 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:54.706636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:54.706638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:54.707257 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:54.707582 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:54.708078 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 10:48:54.708537 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":315,"threads_started":1}
I20260928 10:48:54.710433 26134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:54.795400 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 26011
2026-09-28T10:48:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64441 ms)
[----------] 30 tests from SecurityITest (304345 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:48:54 dist-test-slave-mn38 krb5kdc[26197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:48:54 dist-test-slave-mn38 krb5kdc[26197](info): set up 2 sockets
Sep 28 10:48:54 dist-test-slave-mn38 krb5kdc[26197](info): commencing operation
krb5kdc: starting...
W20260928 10:48:55.842496 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:48:55 dist-test-slave-mn38 krb5kdc[26197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:48:56.095372 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:32895
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:32895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:48:56.448915 26210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:56.449242 26210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:56.449379 26210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:56.457134 26210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:48:56.457267 26210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:56.457319 26210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:56.457398 26210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:48:56.457456 26210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:48:56.468103 26210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:32895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:32895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:56.469535 26210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:56.471235 26210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:56.483804 26215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:56.483752 26216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:56.483763 26218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:56.484223 26210 server_base.cc:1034] running on GCE node
I20260928 10:48:56.485100 26210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:56.486624 26210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:56.487865 26210 hybrid_clock.cc:648] HybridClock initialized: now 1790592536487807 us; error 48 us; skew 500 ppm
I20260928 10:48:56.488241 26210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:56 dist-test-slave-mn38 krb5kdc[26197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592536, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:56.491792 26210 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:48:56.493773 26210 webserver.cc:460] Webserver started at http://127.8.218.254:43527/ using document root <none> and password file <none>
I20260928 10:48:56.494599 26210 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:56.494750 26210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:56.495108 26210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:48:56.498759 26210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "0a86eef5e1ce4d8396e76bbba1a61673"
format_stamp: "Formatted at 2026-09-28 10:48:56 on dist-test-slave-mn38"
server_key: "82ffdbc2e0839a41c49361fd68b62034"
server_key_iv: "f8f1dfdeca53050c93d3484929634ceb"
server_key_version: "encryptionkey@0"
I20260928 10:48:56.499779 26210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "0a86eef5e1ce4d8396e76bbba1a61673"
format_stamp: "Formatted at 2026-09-28 10:48:56 on dist-test-slave-mn38"
server_key: "82ffdbc2e0839a41c49361fd68b62034"
server_key_iv: "f8f1dfdeca53050c93d3484929634ceb"
server_key_version: "encryptionkey@0"
I20260928 10:48:56.506362 26210 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 10:48:56.511615 26225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:56.513644 26210 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:48:56.513870 26210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "0a86eef5e1ce4d8396e76bbba1a61673"
format_stamp: "Formatted at 2026-09-28 10:48:56 on dist-test-slave-mn38"
server_key: "82ffdbc2e0839a41c49361fd68b62034"
server_key_iv: "f8f1dfdeca53050c93d3484929634ceb"
server_key_version: "encryptionkey@0"
I20260928 10:48:56.514102 26210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:56.533133 26210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:56.539197 26210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:56.539556 26210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:56.559460 26210 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:32895
I20260928 10:48:56.559460 26276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:32895 every 8 connection(s)
I20260928 10:48:56.560976 26210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:48:56.561470 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 26210
I20260928 10:48:56.561694 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:48:56.562518 9067 external_mini_cluster.cc:1428] Setting key a8d5f1e8caa9b06beeb94bd7429c0a1e
I20260928 10:48:56.566814 26277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:48:56 dist-test-slave-mn38 krb5kdc[26197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:48:56.580538 26277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673: Bootstrap starting.
I20260928 10:48:56.584813 26277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673: Neither blocks nor log segments found. Creating new log.
I20260928 10:48:56.586177 26277 log.cc:824] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:56.589197 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:56.564674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.8.218.254:32895)
0928 10:48:56.565469 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:56.565485 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:56.565539 (+ 54us) server_negotiation.cc:407] Connection header received
0928 10:48:56.568100 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:56.568129 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:56.568626 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:56.569212 (+ 586us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:56.570872 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:56.572282 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:56.573638 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:56.574367 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:56.574550 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:56.579310 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:56.579323 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:56.579345 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:56.579403 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:56.583626 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:56.584634 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:56.584644 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:56.584653 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:56.584898 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:56.585482 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:56.585491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:56.585494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:56.586302 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:56.586616 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:56.587145 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 10:48:56.587706 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":342,"threads_started":1}
I20260928 10:48:56.590173 26277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673: No bootstrap required, opened a new log
I20260928 10:48:56.599745 26277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a86eef5e1ce4d8396e76bbba1a61673" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32895 } }
I20260928 10:48:56.600195 26277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:48:56.600325 26277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a86eef5e1ce4d8396e76bbba1a61673, State: Initialized, Role: FOLLOWER
I20260928 10:48:56.601058 26277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a86eef5e1ce4d8396e76bbba1a61673" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32895 } }
I20260928 10:48:56.601336 26277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:48:56.601584 26277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:48:56.601768 26277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:48:56.604082 26277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a86eef5e1ce4d8396e76bbba1a61673" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32895 } }
I20260928 10:48:56.604668 26277 leader_election.cc:302] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a86eef5e1ce4d8396e76bbba1a61673; no voters:
I20260928 10:48:56.605486 26277 leader_election.cc:288] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:48:56.605721 26282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:48:56.607072 26282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [term 1 LEADER]: Becoming Leader. State: Replica: 0a86eef5e1ce4d8396e76bbba1a61673, State: Running, Role: LEADER
I20260928 10:48:56.607764 26282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a86eef5e1ce4d8396e76bbba1a61673" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32895 } }
I20260928 10:48:56.608552 26277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:48:56.610878 26284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a86eef5e1ce4d8396e76bbba1a61673. Latest consensus state: current_term: 1 leader_uuid: "0a86eef5e1ce4d8396e76bbba1a61673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a86eef5e1ce4d8396e76bbba1a61673" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32895 } } }
I20260928 10:48:56.611366 26284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [sys.catalog]: This master's current role is: LEADER
I20260928 10:48:56.610828 26283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a86eef5e1ce4d8396e76bbba1a61673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a86eef5e1ce4d8396e76bbba1a61673" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 32895 } } }
I20260928 10:48:56.612174 26283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673 [sys.catalog]: This master's current role is: LEADER
I20260928 10:48:56.613838 26289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:48:56.619654 26289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:48:56.632261 26289 catalog_manager.cc:1347] Generated new cluster ID: 3d213e5420f64f1394c330a4ae897776
I20260928 10:48:56.632457 26289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:48:56.651739 26289 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:48:56.653563 26289 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:48:56.666676 26289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673: Generated new TSK 0
I20260928 10:48:56.667675 26289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:48:56.747113 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32895
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:48:57.093318 26303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:57.093685 26303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:57.093818 26303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:57.102231 26303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:57.102391 26303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:57.102589 26303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:48:57.113634 26303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:57.115027 26303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:57.116443 26303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:57.130220 26309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:57.130692 26311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:57.130721 26308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:57.131049 26303 server_base.cc:1034] running on GCE node
I20260928 10:48:57.131729 26303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:57.132717 26303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:57.133941 26303 hybrid_clock.cc:648] HybridClock initialized: now 1790592537133884 us; error 37 us; skew 500 ppm
I20260928 10:48:57.134277 26303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:57 dist-test-slave-mn38 krb5kdc[26197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:57.137792 26303 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:48:57.139492 26303 webserver.cc:460] Webserver started at http://127.8.218.193:42437/ using document root <none> and password file <none>
I20260928 10:48:57.140329 26303 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:57.140468 26303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:57.140795 26303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:48:57.144080 26303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "3c575eec1c4944ff9b6fe4e4642b932d"
format_stamp: "Formatted at 2026-09-28 10:48:57 on dist-test-slave-mn38"
server_key: "e9f6d626f0a469fd9ac4522f50cd7348"
server_key_iv: "8e04b823d52342ab4ff0a2366e0db3d1"
server_key_version: "encryptionkey@0"
I20260928 10:48:57.144969 26303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "3c575eec1c4944ff9b6fe4e4642b932d"
format_stamp: "Formatted at 2026-09-28 10:48:57 on dist-test-slave-mn38"
server_key: "e9f6d626f0a469fd9ac4522f50cd7348"
server_key_iv: "8e04b823d52342ab4ff0a2366e0db3d1"
server_key_version: "encryptionkey@0"
I20260928 10:48:57.151206 26303 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 10:48:57.155280 26318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:57.156947 26303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:48:57.157147 26303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "3c575eec1c4944ff9b6fe4e4642b932d"
format_stamp: "Formatted at 2026-09-28 10:48:57 on dist-test-slave-mn38"
server_key: "e9f6d626f0a469fd9ac4522f50cd7348"
server_key_iv: "8e04b823d52342ab4ff0a2366e0db3d1"
server_key_version: "encryptionkey@0"
I20260928 10:48:57.157387 26303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:57.176409 26303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:57.180734 26303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:57.181010 26303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:57.182073 26303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:57.183432 26303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:57.183573 26303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:57.183701 26303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:57.183755 26303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:57.214711 26303 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:43979
I20260928 10:48:57.214752 26430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:43979 every 8 connection(s)
I20260928 10:48:57.216081 26303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:48:57.222653 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 26303
I20260928 10:48:57.222862 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:48:57.223505 9067 external_mini_cluster.cc:1428] Setting key c3dcfc0cda8e43d7b0ee78057ae75962
Sep 28 10:48:57 dist-test-slave-mn38 krb5kdc[26197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:48:57.245307 26433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:57.219014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:33759 (local address 127.8.218.254:32895)
0928 10:48:57.219582 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:57.219587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:57.221090 (+ 1503us) server_negotiation.cc:407] Connection header received
0928 10:48:57.223503 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:57.223509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:57.223622 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:57.223741 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:57.225942 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:57.226707 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:57.228431 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:57.228879 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:57.229009 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:57.238573 (+ 9564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:57.238580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:57.238583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:57.238627 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:57.241350 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:57.242634 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:57.242640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:57.242642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:57.242778 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:57.243306 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:57.243310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:57.243312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:57.243604 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:57.243720 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:57.244826 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0928 10:48:57.245039 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260928 10:48:57.246824 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:57.219670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:32895 (local address 127.8.218.193:33759)
0928 10:48:57.220794 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:57.220844 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:57.223095 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:57.224076 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:57.224091 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:57.224846 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:48:57.225752 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:57.225777 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:57.226891 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:57.226895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:57.228248 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:57.228265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:57.229176 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:57.229184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:57.229492 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:57.235053 (+ 5561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:48:57.235101 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:48:57.238216 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:48:57.241575 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:57.241588 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:57.241611 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:57.242452 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:57.242895 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:57.242901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:57.242905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:57.243182 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:57.243739 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:48:57.243748 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:48:57.244270 (+ 522us) client_negotiation.cc:769] Sending connection context
0928 10:48:57.244602 (+ 332us) client_negotiation.cc:240] Negotiation successful
0928 10:48:57.245274 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":559,"threads_started":1}
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
I20260928 10:48:57.248674 26431 heartbeater.cc:347] Connected to a master server at 127.8.218.254:32895
I20260928 10:48:57.249074 26431 heartbeater.cc:464] Registering TS with master...
I20260928 10:48:57.249984 26431 heartbeater.cc:511] Master 127.8.218.254:32895 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
I20260928 10:48:57.252516 26242 ts_manager.cc:194] Registered new tserver with Master: 3c575eec1c4944ff9b6fe4e4642b932d (127.8.218.193:43979)
I20260928 10:48:57.254226 26242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:33759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:48:57.280802 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32895
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:48:57.623283 26437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:57.623620 26437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:57.623725 26437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:57.631788 26437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:57.631930 26437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:57.632048 26437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:48:57.642143 26437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:57.643424 26437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:57.644801 26437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:57.658666 26443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:57.658666 26445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:57.658713 26442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:57.659533 26437 server_base.cc:1034] running on GCE node
I20260928 10:48:57.660241 26437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:57.661239 26437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:57.662446 26437 hybrid_clock.cc:648] HybridClock initialized: now 1790592537662373 us; error 62 us; skew 500 ppm
I20260928 10:48:57.662762 26437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:57 dist-test-slave-mn38 krb5kdc[26197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:57.665854 26437 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:48:57.667632 26437 webserver.cc:460] Webserver started at http://127.8.218.194:38539/ using document root <none> and password file <none>
I20260928 10:48:57.668422 26437 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:57.668533 26437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:57.668843 26437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:48:57.672109 26437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "b729f6f62c7d4aadb229854501037ef8"
format_stamp: "Formatted at 2026-09-28 10:48:57 on dist-test-slave-mn38"
server_key: "0055e26e9d0644431d49e83ee08b4ff8"
server_key_iv: "5f885f9775052ed9d3352aa46644845f"
server_key_version: "encryptionkey@0"
I20260928 10:48:57.673005 26437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "b729f6f62c7d4aadb229854501037ef8"
format_stamp: "Formatted at 2026-09-28 10:48:57 on dist-test-slave-mn38"
server_key: "0055e26e9d0644431d49e83ee08b4ff8"
server_key_iv: "5f885f9775052ed9d3352aa46644845f"
server_key_version: "encryptionkey@0"
I20260928 10:48:57.678900 26437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 10:48:57.682615 26452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:57.684140 26437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:48:57.684378 26437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "b729f6f62c7d4aadb229854501037ef8"
format_stamp: "Formatted at 2026-09-28 10:48:57 on dist-test-slave-mn38"
server_key: "0055e26e9d0644431d49e83ee08b4ff8"
server_key_iv: "5f885f9775052ed9d3352aa46644845f"
server_key_version: "encryptionkey@0"
I20260928 10:48:57.684584 26437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:57.714083 26437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:57.718612 26437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:57.718920 26437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:57.719944 26437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:57.721408 26437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:57.721532 26437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 10:48:57.721668 26437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:57.721729 26437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:57.754565 26437 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:37409
I20260928 10:48:57.754612 26564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:37409 every 8 connection(s)
I20260928 10:48:57.755941 26437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:48:57.766053 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 26437
I20260928 10:48:57.766223 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:48:57.766686 9067 external_mini_cluster.cc:1428] Setting key 2a7fc844b72c6e693763c214caa165d2
Sep 28 10:48:57 dist-test-slave-mn38 krb5kdc[26197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:48:57.778573 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:57.758749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:36411 (local address 127.8.218.254:32895)
0928 10:48:57.759250 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:57.759255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:57.760312 (+ 1057us) server_negotiation.cc:407] Connection header received
0928 10:48:57.762633 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:57.762640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:57.762747 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:57.762861 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:57.764791 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:57.765724 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:57.767653 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:57.768344 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:57.768503 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:57.772431 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:57.772437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:57.772440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:57.772483 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:57.775174 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:57.776047 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:57.776053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:57.776059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:57.776200 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:57.776672 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:57.776676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:57.776678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:57.776961 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:57.777110 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:57.777901 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 10:48:57.778168 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":276,"threads_started":1}
I20260928 10:48:57.779644 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:57.759261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:32895 (local address 127.8.218.194:36411)
0928 10:48:57.760096 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:57.760151 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:57.762247 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:57.763182 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:57.763193 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:57.763828 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:48:57.764530 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:57.764554 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:57.765933 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:57.765938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:57.767442 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:57.767465 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:57.768627 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:57.768635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:57.768974 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:57.769716 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:48:57.769749 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:48:57.772119 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:48:57.775386 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:57.775394 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:57.775410 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:57.775901 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:57.776330 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:57.776334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:57.776337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:57.776553 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:57.777095 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:48:57.777105 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:48:57.777522 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 10:48:57.777797 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 10:48:57.778381 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":342,"threads_started":1}
I20260928 10:48:57.781200 26565 heartbeater.cc:347] Connected to a master server at 127.8.218.254:32895
I20260928 10:48:57.781581 26565 heartbeater.cc:464] Registering TS with master...
I20260928 10:48:57.782492 26565 heartbeater.cc:511] Master 127.8.218.254:32895 requested a full tablet report, sending...
I20260928 10:48:57.784291 26242 ts_manager.cc:194] Registered new tserver with Master: b729f6f62c7d4aadb229854501037ef8 (127.8.218.194:37409)
I20260928 10:48:57.785127 26242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:36411
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:48:57.821975 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32895
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:48:58.161777 26571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:58.162096 26571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:58.162250 26571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:58.169884 26571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:58.170006 26571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:58.170171 26571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:48:58.180130 26571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:32895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:58.181469 26571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:58.182801 26571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:58.196065 26576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:58.196028 26577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:58.196105 26579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:58.197058 26571 server_base.cc:1034] running on GCE node
I20260928 10:48:58.197841 26571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:58.198896 26571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:58.200109 26571 hybrid_clock.cc:648] HybridClock initialized: now 1790592538200070 us; error 39 us; skew 500 ppm
I20260928 10:48:58.200441 26571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:48:58 dist-test-slave-mn38 krb5kdc[26197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592538, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:48:58.204010 26571 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:48:58.205868 26571 webserver.cc:460] Webserver started at http://127.8.218.195:43891/ using document root <none> and password file <none>
I20260928 10:48:58.206616 26571 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:58.206748 26571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:58.207067 26571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:48:58.210387 26571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "0347854fd2394555986e96f04a431d29"
format_stamp: "Formatted at 2026-09-28 10:48:58 on dist-test-slave-mn38"
server_key: "975f63c7b4d91a86f77b1baa7d375cb6"
server_key_iv: "faaa4b9fa2df8cd70cc36b25d17df4e7"
server_key_version: "encryptionkey@0"
I20260928 10:48:58.211252 26571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "0347854fd2394555986e96f04a431d29"
format_stamp: "Formatted at 2026-09-28 10:48:58 on dist-test-slave-mn38"
server_key: "975f63c7b4d91a86f77b1baa7d375cb6"
server_key_iv: "faaa4b9fa2df8cd70cc36b25d17df4e7"
server_key_version: "encryptionkey@0"
I20260928 10:48:58.217844 26571 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 10:48:58.222807 26586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:58.224494 26571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:48:58.224674 26571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "0347854fd2394555986e96f04a431d29"
format_stamp: "Formatted at 2026-09-28 10:48:58 on dist-test-slave-mn38"
server_key: "975f63c7b4d91a86f77b1baa7d375cb6"
server_key_iv: "faaa4b9fa2df8cd70cc36b25d17df4e7"
server_key_version: "encryptionkey@0"
I20260928 10:48:58.224921 26571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:58.251722 26571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:58.256305 26571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:58.256558 26571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:58.257550 26571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:48:58.258299 26431 heartbeater.cc:503] Master 127.8.218.254:32895 was elected leader, sending a full tablet report...
I20260928 10:48:58.259042 26571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:48:58.259121 26571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:58.259271 26571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:48:58.259325 26571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:58.290733 26571 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:42683
I20260928 10:48:58.290777 26698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:42683 every 8 connection(s)
I20260928 10:48:58.292037 26571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:48:58.296896 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 26571
I20260928 10:48:58.297087 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:48:58.297688 9067 external_mini_cluster.cc:1428] Setting key bd7549ed9ef330acdd513180571d769c
Sep 28 10:48:58 dist-test-slave-mn38 krb5kdc[26197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592538, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:48:58.314481 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.294592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:57957 (local address 127.8.218.254:32895)
0928 10:48:58.295073 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.295077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.296145 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 10:48:58.298313 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.298318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.298423 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.298568 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:58.300823 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.301864 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.303605 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.304011 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.304112 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.307831 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:58.307836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:58.307839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:58.307884 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:58.310629 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.311509 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.311515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.311518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.311682 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.312284 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.312289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.312292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.312673 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:58.312843 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.313832 (+ 989us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.314084 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":251,"threads_started":1}
I20260928 10:48:58.315718 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.295171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:32895 (local address 127.8.218.195:57957)
0928 10:48:58.295952 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:58.296007 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:58.297943 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:58.298884 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:58.298898 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:58.299660 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:48:58.300565 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.300596 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.302041 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.302046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.303441 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.303457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.304294 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.304305 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.304539 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.305256 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:48:58.305282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:48:58.307578 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:48:58.310794 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:58.310803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:58.310820 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:58.311383 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:58.311826 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:48:58.311830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:48:58.311832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:48:58.312129 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:48:58.312811 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:48:58.312818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:48:58.313348 (+ 530us) client_negotiation.cc:769] Sending connection context
0928 10:48:58.313729 (+ 381us) client_negotiation.cc:240] Negotiation successful
0928 10:48:58.314345 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":301,"threads_started":1}
I20260928 10:48:58.317233 26699 heartbeater.cc:347] Connected to a master server at 127.8.218.254:32895
I20260928 10:48:58.317608 26699 heartbeater.cc:464] Registering TS with master...
I20260928 10:48:58.318557 26699 heartbeater.cc:511] Master 127.8.218.254:32895 requested a full tablet report, sending...
I20260928 10:48:58.320179 26242 ts_manager.cc:194] Registered new tserver with Master: 0347854fd2394555986e96f04a431d29 (127.8.218.195:42683)
I20260928 10:48:58.321041 26242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:57957
I20260928 10:48:58.326838 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:48:58.344102 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.331355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.8.218.254:32895)
0928 10:48:58.331545 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.331552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.331906 (+ 354us) server_negotiation.cc:407] Connection header received
0928 10:48:58.332079 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.332085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.332168 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.332253 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:58.333706 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.334433 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.335752 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.336197 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.336311 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.338720 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:58.338725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:58.338728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:58.338775 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:58.341375 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.342097 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.342103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.342106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.342235 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.342608 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.342612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.342614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.342965 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:58.343193 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.343550 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.343793 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:48:58.348413 26242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:48:58.356319 26242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:48:58.380623 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.372004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.8.218.193:43979)
0928 10:48:58.373058 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.373065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.373092 (+ 27us) server_negotiation.cc:407] Connection header received
0928 10:48:58.373184 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.373191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.373423 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.373624 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:58.374570 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.376077 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.378248 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.379534 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.379659 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.379893 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.380140 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.380314 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":900,"threads_started":1}
I20260928 10:48:58.380733 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.371146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:43979 (local address 127.0.0.1:39378)
0928 10:48:58.372530 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:58.372544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:58.372644 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:58.373900 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:58.373904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:58.373925 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:58.374401 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.374414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.376265 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.376269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.378084 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.378102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.379725 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.379731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.379917 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.379968 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 10:48:58.380083 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 10:48:58.380279 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1280,"mutex_wait_us":63,"thread_start_us":589,"threads_started":1}
I20260928 10:48:58.381191 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.370395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:37409 (local address 127.0.0.1:52236)
0928 10:48:58.371189 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:58.371238 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:58.371418 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:58.373651 (+ 2233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:58.373655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:58.373702 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:58.374302 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.374312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.376201 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.376205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.378505 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.378520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.380327 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.380333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.380515 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.380547 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:48:58.380620 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 10:48:58.380762 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":50,"spinlock_wait_cycles":9600,"thread_start_us":2076,"threads_started":1}
I20260928 10:48:58.381942 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.370466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:42683 (local address 127.0.0.1:34490)
0928 10:48:58.371982 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:58.371998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:58.372091 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:58.373642 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:58.373649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:58.373691 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:58.374166 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.374180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.375971 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.375977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.379094 (+ 3117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.379106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.381175 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.381182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.381333 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.381363 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 10:48:58.381458 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 10:48:58.381579 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1409,"mutex_wait_us":133,"spinlock_wait_cycles":1920,"thread_start_us":1839,"threads_started":1}
I20260928 10:48:58.382237 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.370977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.8.218.194:37409)
0928 10:48:58.373111 (+ 2134us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.373118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.373139 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:48:58.373250 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.373256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.373423 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.373989 (+ 566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:58.374476 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.376068 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.378697 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.380081 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.381529 (+ 1448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.381705 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.381778 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.381932 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1969,"thread_start_us":2103,"threads_started":1}
I20260928 10:48:58.382746 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.371419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.8.218.195:42683)
0928 10:48:58.373025 (+ 1606us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.373050 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.373084 (+ 34us) server_negotiation.cc:407] Connection header received
0928 10:48:58.373181 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.373189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.373408 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.373609 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:58.374389 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.375855 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.379270 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.380706 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.380816 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.381029 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.382276 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.382440 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451,"thread_start_us":1306,"threads_started":1}
I20260928 10:48:58.384713 26366 tablet_service.cc:1467] Processing CreateTablet for tablet 92ac652c6d794348a88e9a1ee956c381 (DEFAULT_TABLE table=test-table [id=6bbd033ca5de49debda9ebbcc45b44ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:48:58.385980 26366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ac652c6d794348a88e9a1ee956c381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:48:58.385774 26500 tablet_service.cc:1467] Processing CreateTablet for tablet 92ac652c6d794348a88e9a1ee956c381 (DEFAULT_TABLE table=test-table [id=6bbd033ca5de49debda9ebbcc45b44ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:48:58.386391 26634 tablet_service.cc:1467] Processing CreateTablet for tablet 92ac652c6d794348a88e9a1ee956c381 (DEFAULT_TABLE table=test-table [id=6bbd033ca5de49debda9ebbcc45b44ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:48:58.387132 26500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ac652c6d794348a88e9a1ee956c381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:48:58.387727 26634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ac652c6d794348a88e9a1ee956c381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:48:58.398910 26718 tablet_bootstrap.cc:492] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Bootstrap starting.
I20260928 10:48:58.403468 26718 tablet_bootstrap.cc:654] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Neither blocks nor log segments found. Creating new log.
I20260928 10:48:58.404739 26718 log.cc:824] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:58.407143 26720 tablet_bootstrap.cc:492] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: Bootstrap starting.
I20260928 10:48:58.408282 26718 tablet_bootstrap.cc:492] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: No bootstrap required, opened a new log
I20260928 10:48:58.408617 26718 ts_tablet_manager.cc:1397] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260928 10:48:58.409230 26719 tablet_bootstrap.cc:492] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Bootstrap starting.
I20260928 10:48:58.412698 26719 tablet_bootstrap.cc:654] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Neither blocks nor log segments found. Creating new log.
I20260928 10:48:58.413966 26719 log.cc:824] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:58.416275 26720 tablet_bootstrap.cc:654] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: Neither blocks nor log segments found. Creating new log.
I20260928 10:48:58.416922 26718 raft_consensus.cc:348] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.417378 26718 raft_consensus.cc:374] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:48:58.417544 26718 raft_consensus.cc:729] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c575eec1c4944ff9b6fe4e4642b932d, State: Initialized, Role: FOLLOWER
I20260928 10:48:58.417583 26719 tablet_bootstrap.cc:492] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: No bootstrap required, opened a new log
I20260928 10:48:58.417697 26720 log.cc:824] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:58.417896 26719 ts_tablet_manager.cc:1397] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 10:48:58.418390 26718 consensus_queue.cc:260] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.420295 26718 ts_tablet_manager.cc:1428] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 10:48:58.421234 26720 tablet_bootstrap.cc:492] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: No bootstrap required, opened a new log
I20260928 10:48:58.421531 26720 ts_tablet_manager.cc:1397] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20260928 10:48:58.428038 26719 raft_consensus.cc:348] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.428474 26719 raft_consensus.cc:374] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:48:58.428579 26719 raft_consensus.cc:729] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b729f6f62c7d4aadb229854501037ef8, State: Initialized, Role: FOLLOWER
I20260928 10:48:58.429232 26719 consensus_queue.cc:260] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.429935 26720 raft_consensus.cc:348] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.430279 26720 raft_consensus.cc:374] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:48:58.430373 26720 raft_consensus.cc:729] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0347854fd2394555986e96f04a431d29, State: Initialized, Role: FOLLOWER
I20260928 10:48:58.430477 26565 heartbeater.cc:503] Master 127.8.218.254:32895 was elected leader, sending a full tablet report...
I20260928 10:48:58.430974 26719 ts_tablet_manager.cc:1428] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260928 10:48:58.431000 26720 consensus_queue.cc:260] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.432042 26699 heartbeater.cc:503] Master 127.8.218.254:32895 was elected leader, sending a full tablet report...
I20260928 10:48:58.432574 26720 ts_tablet_manager.cc:1428] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 10:48:58.448351 26726 raft_consensus.cc:482] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:48:58.448648 26726 raft_consensus.cc:504] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.450526 26726 leader_election.cc:288] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b729f6f62c7d4aadb229854501037ef8 (127.8.218.194:37409), 3c575eec1c4944ff9b6fe4e4642b932d (127.8.218.193:43979)
I20260928 10:48:58.458274 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.451138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:55605 (local address 127.8.218.193:43979)
0928 10:48:58.451360 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.451366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.451389 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:48:58.451919 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.451925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.452011 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.452112 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:58.452517 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.453544 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.456047 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.457274 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.457492 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.457580 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.457882 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.458016 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 10:48:58.458324 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.451790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:32857 (local address 127.8.218.194:37409)
0928 10:48:58.451988 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.451994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.452014 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:48:58.452208 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.452215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.452298 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.452357 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:48:58.453000 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.453937 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.456241 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.457490 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.457639 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.457705 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.457977 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.458102 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:48:58.458323 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.450857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:43979 (local address 127.8.218.195:55605)
0928 10:48:58.451198 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:58.451221 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:58.451377 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:58.452159 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:58.452164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:58.452195 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:58.452366 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.452375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.453707 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.453713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.455852 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.455869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.457480 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.457487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.457689 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.457736 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 10:48:58.457837 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 10:48:58.457994 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":65}
I20260928 10:48:58.458411 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.450928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:37409 (local address 127.8.218.195:32857)
0928 10:48:58.451899 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 10:48:58.451915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 10:48:58.452031 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:48:58.452516 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:48:58.452519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:48:58.452540 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:48:58.452840 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.452848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.454075 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.454080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.456075 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:48:58.456085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.457678 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.457684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:48:58.457839 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.457867 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 10:48:58.457937 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 10:48:58.458047 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":3,"thread_start_us":636,"threads_started":1}
I20260928 10:48:58.459575 26386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ac652c6d794348a88e9a1ee956c381" candidate_uuid: "0347854fd2394555986e96f04a431d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" is_pre_election: true
I20260928 10:48:58.459915 26520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ac652c6d794348a88e9a1ee956c381" candidate_uuid: "0347854fd2394555986e96f04a431d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b729f6f62c7d4aadb229854501037ef8" is_pre_election: true
I20260928 10:48:58.460124 26386 raft_consensus.cc:2454] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0347854fd2394555986e96f04a431d29 in term 0.
I20260928 10:48:58.460371 26520 raft_consensus.cc:2454] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0347854fd2394555986e96f04a431d29 in term 0.
I20260928 10:48:58.460884 26588 leader_election.cc:302] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0347854fd2394555986e96f04a431d29, 3c575eec1c4944ff9b6fe4e4642b932d; no voters:
I20260928 10:48:58.461306 26726 raft_consensus.cc:2781] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:48:58.461467 26726 raft_consensus.cc:482] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:48:58.461575 26726 raft_consensus.cc:3037] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:48:58.464018 26726 raft_consensus.cc:504] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.464710 26726 leader_election.cc:288] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [CANDIDATE]: Term 1 election: Requested vote from peers b729f6f62c7d4aadb229854501037ef8 (127.8.218.194:37409), 3c575eec1c4944ff9b6fe4e4642b932d (127.8.218.193:43979)
I20260928 10:48:58.465252 26520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ac652c6d794348a88e9a1ee956c381" candidate_uuid: "0347854fd2394555986e96f04a431d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b729f6f62c7d4aadb229854501037ef8"
I20260928 10:48:58.465307 26386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ac652c6d794348a88e9a1ee956c381" candidate_uuid: "0347854fd2394555986e96f04a431d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c575eec1c4944ff9b6fe4e4642b932d"
I20260928 10:48:58.465472 26520 raft_consensus.cc:3037] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:48:58.465543 26386 raft_consensus.cc:3037] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:48:58.467720 26386 raft_consensus.cc:2454] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0347854fd2394555986e96f04a431d29 in term 1.
I20260928 10:48:58.467720 26520 raft_consensus.cc:2454] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0347854fd2394555986e96f04a431d29 in term 1.
I20260928 10:48:58.468140 26588 leader_election.cc:302] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0347854fd2394555986e96f04a431d29, 3c575eec1c4944ff9b6fe4e4642b932d; no voters:
I20260928 10:48:58.468477 26726 raft_consensus.cc:2781] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 1 FOLLOWER]: Leader election won for term 1
W20260928 10:48:58.468696 26432 tablet.cc:2369] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:48:58.468982 26726 raft_consensus.cc:686] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 1 LEADER]: Becoming Leader. State: Replica: 0347854fd2394555986e96f04a431d29, State: Running, Role: LEADER
I20260928 10:48:58.469587 26726 consensus_queue.cc:237] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } }
I20260928 10:48:58.474255 26242 catalog_manager.cc:5495] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0347854fd2394555986e96f04a431d29 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "0347854fd2394555986e96f04a431d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0347854fd2394555986e96f04a431d29" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 } health_report { overall_health: UNKNOWN } } }
W20260928 10:48:58.508016 26566 tablet.cc:2369] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:48:58.510433 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.504400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.8.218.195:42683)
0928 10:48:58.504635 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.504640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.504660 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:48:58.504882 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.504888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.504995 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.505144 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:48:58.505582 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.506450 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.508069 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.508535 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.508665 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.509231 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:48:58.509839 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:48:58.509981 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.510073 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.510165 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 10:48:58.520231 26520 raft_consensus.cc:1261] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 1 FOLLOWER]: Refusing update from remote peer 0347854fd2394555986e96f04a431d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:48:58.520222 26386 raft_consensus.cc:1261] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 1 FOLLOWER]: Refusing update from remote peer 0347854fd2394555986e96f04a431d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:48:58.521516 26728 consensus_queue.cc:1035] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b729f6f62c7d4aadb229854501037ef8" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:48:58.521910 26726 consensus_queue.cc:1035] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c575eec1c4944ff9b6fe4e4642b932d" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:48:58.536299 26732 mvcc.cc:205] Tried to move back new op lower bound from 7334267037766479872 to 7334267037575634944. Current Snapshot: MvccSnapshot[applied={T|T < 7334267037766479872}]
I20260928 10:48:58.551442 26242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:48:58.551679 26242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:48:58.555521 26242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a86eef5e1ce4d8396e76bbba1a61673: Sending DeleteTablet for 3 replicas of tablet 92ac652c6d794348a88e9a1ee956c381
I20260928 10:48:58.556931 26500 tablet_service.cc:1514] Processing DeleteTablet for tablet 92ac652c6d794348a88e9a1ee956c381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:48:58 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:52236
I20260928 10:48:58.557085 26366 tablet_service.cc:1514] Processing DeleteTablet for tablet 92ac652c6d794348a88e9a1ee956c381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:48:58 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:39378
I20260928 10:48:58.557117 26634 tablet_service.cc:1514] Processing DeleteTablet for tablet 92ac652c6d794348a88e9a1ee956c381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:48:58 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:34490
I20260928 10:48:58.559777 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 26303
I20260928 10:48:58.569558 26742 tablet_replica.cc:330] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: stopping tablet replica
I20260928 10:48:58.570070 26742 raft_consensus.cc:2229] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:48:58.570650 26742 raft_consensus.cc:2258] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:48:58.573216 26742 ts_tablet_manager.cc:1905] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:48:58.577875 26741 tablet_replica.cc:330] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: stopping tablet replica
I20260928 10:48:58.578332 26741 raft_consensus.cc:2229] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:48:58.579039 26741 raft_consensus.cc:2258] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:48:58.580327 26742 ts_tablet_manager.cc:1918] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:48:58.580505 26742 log.cc:1196] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/92ac652c6d794348a88e9a1ee956c381
I20260928 10:48:58.581018 26742 ts_tablet_manager.cc:1939] T 92ac652c6d794348a88e9a1ee956c381 P b729f6f62c7d4aadb229854501037ef8: Deleting consensus metadata
I20260928 10:48:58.581972 26740 tablet_replica.cc:330] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: stopping tablet replica
I20260928 10:48:58.582432 26740 raft_consensus.cc:2229] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:48:58.582870 26741 ts_tablet_manager.cc:1905] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:48:58.582948 26740 raft_consensus.cc:2258] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:48:58.584940 26740 ts_tablet_manager.cc:1905] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 10:48:58 dist-test-slave-mn38 krb5kdc[26197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:48:58.591070 26741 ts_tablet_manager.cc:1918] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:48:58.591328 26741 log.cc:1196] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/92ac652c6d794348a88e9a1ee956c381
I20260928 10:48:58.591929 26741 ts_tablet_manager.cc:1939] T 92ac652c6d794348a88e9a1ee956c381 P 0347854fd2394555986e96f04a431d29: Deleting consensus metadata
I20260928 10:48:58.591955 26740 ts_tablet_manager.cc:1918] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:48:58.593854 26226 catalog_manager.cc:4841] TS 0347854fd2394555986e96f04a431d29 (127.8.218.195:42683): tablet 92ac652c6d794348a88e9a1ee956c381 (table test-table [id=6bbd033ca5de49debda9ebbcc45b44ba]) successfully deleted
I20260928 10:48:58.594079 26740 log.cc:1196] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/92ac652c6d794348a88e9a1ee956c381
I20260928 10:48:58.593832 26229 catalog_manager.cc:4841] TS b729f6f62c7d4aadb229854501037ef8 (127.8.218.194:37409): tablet 92ac652c6d794348a88e9a1ee956c381 (table test-table [id=6bbd033ca5de49debda9ebbcc45b44ba]) successfully deleted
I20260928 10:48:58.594630 26740 ts_tablet_manager.cc:1939] T 92ac652c6d794348a88e9a1ee956c381 P 3c575eec1c4944ff9b6fe4e4642b932d: Deleting consensus metadata
I20260928 10:48:58.596593 26227 catalog_manager.cc:4841] TS 3c575eec1c4944ff9b6fe4e4642b932d (127.8.218.193:43979): tablet 92ac652c6d794348a88e9a1ee956c381 (table test-table [id=6bbd033ca5de49debda9ebbcc45b44ba]) successfully deleted
I20260928 10:48:58.599931 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.561041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.8.218.193:43979)
0928 10:48:58.561223 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.561230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.578998 (+ 17768us) server_negotiation.cc:407] Connection header received
0928 10:48:58.579542 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.579548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.579631 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.579698 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:58.581631 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.582838 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.584793 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.585579 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.585679 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.591502 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:58.591523 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:58.591545 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:58.591606 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:58.595964 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.597427 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.597481 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.597485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.597686 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.598290 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.598295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.598297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.598701 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:58.598949 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.599393 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.599607 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:48:58.600757 26426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:58.712819 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 26303
I20260928 10:48:58.729135 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 26437
Sep 28 10:48:58 dist-test-slave-mn38 krb5kdc[26197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:48:58.747205 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.730181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.8.218.194:37409)
0928 10:48:58.730344 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.730348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.730366 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:48:58.730421 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.730425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.730488 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.730568 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:58.732309 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.733101 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.735279 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.735734 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.735844 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.741189 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:58.741212 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:58.741228 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:58.741274 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:58.744217 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.745147 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.745155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.745157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.745294 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.745718 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.745721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.745722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.746122 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:58.746354 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.746737 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.746928 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:48:58.747901 26560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:58.843636 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 26437
I20260928 10:48:58.860739 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 26571
Sep 28 10:48:58 dist-test-slave-mn38 krb5kdc[26197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:48:58.879392 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:58.861706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.8.218.195:42683)
0928 10:48:58.861878 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:58.861883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:58.861901 (+ 18us) server_negotiation.cc:407] Connection header received
0928 10:48:58.862032 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:58.862036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:58.862096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:58.862177 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:58.863897 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.864741 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.866761 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:58.867222 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:58.867320 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:58.873148 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:58.873165 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:58.873181 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:48:58.873230 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:58.876153 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.877066 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.877073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.877076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.877283 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:48:58.877814 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:48:58.877820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:48:58.877822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:48:58.878258 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:58.878521 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:58.878851 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 10:48:58.879022 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:48:58.880234 26694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:59.002311 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 26571
I20260928 10:48:59.018401 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 26210
I20260928 10:48:59.019627 26272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:48:59.132141 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 26210
2026-09-28T10:48:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4344 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T10:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:48:59Z Disabled control of system clock
I20260928 10:48:59.209677 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45079
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:45079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:48:59.561691 26751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:48:59.562026 26751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:48:59.562156 26751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:48:59.569751 26751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:48:59.569876 26751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:48:59.569923 26751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:48:59.570012 26751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:48:59.570071 26751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:48:59.579772 26751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:45079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:48:59.581104 26751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:48:59.582917 26751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:48:59.595145 26756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:59.595216 26757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:48:59.595247 26759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:48:59.596130 26751 server_base.cc:1034] running on GCE node
I20260928 10:48:59.597074 26751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:48:59.598778 26751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:48:59.600054 26751 hybrid_clock.cc:648] HybridClock initialized: now 1790592539599973 us; error 73 us; skew 500 ppm
I20260928 10:48:59.600455 26751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:48:59.603154 26751 webserver.cc:460] Webserver started at http://127.8.218.254:39275/ using document root <none> and password file <none>
I20260928 10:48:59.603953 26751 fs_manager.cc:359] Metadata directory not provided
I20260928 10:48:59.604065 26751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:48:59.604398 26751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:48:59.607733 26751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "51511aa8a7e340bab1612eb4b5d796f4"
format_stamp: "Formatted at 2026-09-28 10:48:59 on dist-test-slave-mn38"
server_key: "2df1fdfa7b42113be976dc90720d7a3e"
server_key_iv: "de47cfbe69f511c56daf17c492a5cff7"
server_key_version: "encryptionkey@0"
I20260928 10:48:59.608896 26751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "51511aa8a7e340bab1612eb4b5d796f4"
format_stamp: "Formatted at 2026-09-28 10:48:59 on dist-test-slave-mn38"
server_key: "2df1fdfa7b42113be976dc90720d7a3e"
server_key_iv: "de47cfbe69f511c56daf17c492a5cff7"
server_key_version: "encryptionkey@0"
I20260928 10:48:59.615196 26751 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.005s
I20260928 10:48:59.619303 26765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:48:59.620949 26751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:48:59.621160 26751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "51511aa8a7e340bab1612eb4b5d796f4"
format_stamp: "Formatted at 2026-09-28 10:48:59 on dist-test-slave-mn38"
server_key: "2df1fdfa7b42113be976dc90720d7a3e"
server_key_iv: "de47cfbe69f511c56daf17c492a5cff7"
server_key_version: "encryptionkey@0"
I20260928 10:48:59.621421 26751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:48:59.639371 26751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:48:59.640304 26751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:48:59.640625 26751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:48:59.658478 26751 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:45079
I20260928 10:48:59.658528 26816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:45079 every 8 connection(s)
I20260928 10:48:59.659962 26751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:48:59.663791 26817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:48:59.664768 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 26751
I20260928 10:48:59.664978 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:48:59.665426 9067 external_mini_cluster.cc:1428] Setting key 07dbd7d051683b11c35cf6ba58275014
I20260928 10:48:59.676857 26817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4: Bootstrap starting.
I20260928 10:48:59.679091 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:48:59.667420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.8.218.254:45079)
0928 10:48:59.668498 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0928 10:48:59.668519 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 10:48:59.668561 (+ 42us) server_negotiation.cc:407] Connection header received
0928 10:48:59.670463 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:48:59.670489 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:48:59.671012 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:48:59.671428 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:48:59.672053 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:59.673352 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:59.674732 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:48:59.675441 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:48:59.675608 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:48:59.676768 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:48:59.676775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:48:59.676791 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:48:59.677374 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:48:59.677492 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:48:59.677516 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:48:59.677622 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 10:48:59.677832 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 10:48:59.678135 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":383,"threads_started":1}
I20260928 10:48:59.680385 26817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4: Neither blocks nor log segments found. Creating new log.
I20260928 10:48:59.681424 26817 log.cc:824] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:48:59.684301 26817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4: No bootstrap required, opened a new log
I20260928 10:48:59.691725 26817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51511aa8a7e340bab1612eb4b5d796f4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45079 } }
I20260928 10:48:59.692155 26817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:48:59.692255 26817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51511aa8a7e340bab1612eb4b5d796f4, State: Initialized, Role: FOLLOWER
I20260928 10:48:59.692878 26817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51511aa8a7e340bab1612eb4b5d796f4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45079 } }
I20260928 10:48:59.693120 26817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:48:59.693256 26817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:48:59.693480 26817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:48:59.695856 26817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51511aa8a7e340bab1612eb4b5d796f4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45079 } }
I20260928 10:48:59.696448 26817 leader_election.cc:302] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51511aa8a7e340bab1612eb4b5d796f4; no voters:
I20260928 10:48:59.697088 26817 leader_election.cc:288] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:48:59.697242 26822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:48:59.698388 26822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [term 1 LEADER]: Becoming Leader. State: Replica: 51511aa8a7e340bab1612eb4b5d796f4, State: Running, Role: LEADER
I20260928 10:48:59.698975 26822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51511aa8a7e340bab1612eb4b5d796f4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45079 } }
I20260928 10:48:59.699532 26817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:48:59.701884 26823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51511aa8a7e340bab1612eb4b5d796f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51511aa8a7e340bab1612eb4b5d796f4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45079 } } }
I20260928 10:48:59.702354 26823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:48:59.702198 26824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51511aa8a7e340bab1612eb4b5d796f4. Latest consensus state: current_term: 1 leader_uuid: "51511aa8a7e340bab1612eb4b5d796f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51511aa8a7e340bab1612eb4b5d796f4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 45079 } } }
I20260928 10:48:59.702476 26824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:48:59.704624 26831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:48:59.710338 26831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:48:59.721189 26831 catalog_manager.cc:1347] Generated new cluster ID: 9e5d00a6879e432d9153a7dad8870f0f
I20260928 10:48:59.721329 26831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:48:59.750077 26831 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:48:59.751506 26831 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:48:59.760440 26831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4: Generated new TSK 0
I20260928 10:48:59.761695 26831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:48:59.766057 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45079
--builtin_ntp_servers=127.8.218.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:49:00.121599 26841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:00.121948 26841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:00.122090 26841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:00.130092 26841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:00.130227 26841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:00.130379 26841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:49:00.140146 26841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:00.141556 26841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:00.143455 26841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:00.157729 26847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:00.157661 26849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:00.157620 26846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:00.158689 26841 server_base.cc:1034] running on GCE node
I20260928 10:49:00.159400 26841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:00.160460 26841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:00.161671 26841 hybrid_clock.cc:648] HybridClock initialized: now 1790592540161621 us; error 48 us; skew 500 ppm
I20260928 10:49:00.162004 26841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:00.164350 26841 webserver.cc:460] Webserver started at http://127.8.218.193:37159/ using document root <none> and password file <none>
I20260928 10:49:00.165104 26841 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:00.165213 26841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:00.165575 26841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:00.168500 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "601ea6e87afb4b699988f9044a821a33"
format_stamp: "Formatted at 2026-09-28 10:49:00 on dist-test-slave-mn38"
server_key: "6ffcb9e1d9b7b1d00112943cd4cab56d"
server_key_iv: "cf63ad976fb23961cbabb1c65e0268e2"
server_key_version: "encryptionkey@0"
I20260928 10:49:00.169324 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "601ea6e87afb4b699988f9044a821a33"
format_stamp: "Formatted at 2026-09-28 10:49:00 on dist-test-slave-mn38"
server_key: "6ffcb9e1d9b7b1d00112943cd4cab56d"
server_key_iv: "cf63ad976fb23961cbabb1c65e0268e2"
server_key_version: "encryptionkey@0"
I20260928 10:49:00.174719 26841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 10:49:00.178920 26855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:00.180546 26841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:49:00.180727 26841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "601ea6e87afb4b699988f9044a821a33"
format_stamp: "Formatted at 2026-09-28 10:49:00 on dist-test-slave-mn38"
server_key: "6ffcb9e1d9b7b1d00112943cd4cab56d"
server_key_iv: "cf63ad976fb23961cbabb1c65e0268e2"
server_key_version: "encryptionkey@0"
I20260928 10:49:00.181000 26841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:00.225507 26841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:00.226491 26841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:00.226801 26841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:00.227880 26841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:00.229576 26841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:00.229682 26841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:00.229805 26841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:00.229878 26841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:00.264452 26841 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:41663
I20260928 10:49:00.264506 26967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:41663 every 8 connection(s)
I20260928 10:49:00.266247 26841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:49:00.274291 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 26841
I20260928 10:49:00.274483 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:49:00.274907 9067 external_mini_cluster.cc:1428] Setting key 45d693cbf39d9bfa2b38be16fee09f47
I20260928 10:49:00.277755 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45079
--builtin_ntp_servers=127.8.218.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:49:00.279726 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:00.269222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:45651 (local address 127.8.218.254:45079)
0928 10:49:00.269794 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:00.269798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:00.271045 (+ 1247us) server_negotiation.cc:407] Connection header received
0928 10:49:00.272869 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:00.272876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:00.272964 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:00.273063 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:00.274347 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:00.275190 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:00.276786 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:00.277202 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:00.277301 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:00.278957 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:00.278962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:00.278965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:00.278983 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:00.279025 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:00.279037 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:00.279109 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:00.279357 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 10:49:00.279447 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":287,"threads_started":1}
I20260928 10:49:00.281082 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:00.269894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45079 (local address 127.8.218.193:45651)
0928 10:49:00.270824 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:00.270907 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:00.272570 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:00.273302 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:00.273311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:00.273413 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:00.274143 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:00.274170 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:00.275368 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:00.275373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:00.276656 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:00.276670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:00.277476 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:00.277486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:00.277798 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:00.278575 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:00.278579 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:00.278635 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:00.278750 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:00.279125 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:00.279134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:00.279142 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:49:00.279318 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 10:49:00.279739 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":276,"threads_started":1}
I20260928 10:49:00.282758 26968 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45079
I20260928 10:49:00.283121 26968 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:00.284066 26968 heartbeater.cc:511] Master 127.8.218.254:45079 requested a full tablet report, sending...
I20260928 10:49:00.286656 26782 ts_manager.cc:194] Registered new tserver with Master: 601ea6e87afb4b699988f9044a821a33 (127.8.218.193:41663)
I20260928 10:49:00.288340 26782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.193:45651
W20260928 10:49:00.628472 26972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:00.628793 26972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:00.628921 26972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:00.636549 26972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:00.636700 26972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:00.636853 26972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:49:00.646448 26972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:00.647787 26972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:00.649684 26972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:00.662341 26978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:00.662809 26977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:00.663225 26980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:00.663998 26972 server_base.cc:1034] running on GCE node
I20260928 10:49:00.664659 26972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:00.665747 26972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:00.666996 26972 hybrid_clock.cc:648] HybridClock initialized: now 1790592540666927 us; error 60 us; skew 500 ppm
I20260928 10:49:00.667371 26972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:00.670039 26972 webserver.cc:460] Webserver started at http://127.8.218.194:35449/ using document root <none> and password file <none>
I20260928 10:49:00.670758 26972 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:00.670864 26972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:00.671175 26972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:00.674084 26972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "45690be67bb340598f2d0825368101af"
format_stamp: "Formatted at 2026-09-28 10:49:00 on dist-test-slave-mn38"
server_key: "dcfa0195345df9f3ea52b614ce216eff"
server_key_iv: "977810bc9d0d9b60ae691e3618a40db4"
server_key_version: "encryptionkey@0"
I20260928 10:49:00.674901 26972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "45690be67bb340598f2d0825368101af"
format_stamp: "Formatted at 2026-09-28 10:49:00 on dist-test-slave-mn38"
server_key: "dcfa0195345df9f3ea52b614ce216eff"
server_key_iv: "977810bc9d0d9b60ae691e3618a40db4"
server_key_version: "encryptionkey@0"
I20260928 10:49:00.680286 26972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 10:49:00.683889 26986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:00.685263 26972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 10:49:00.685415 26972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "45690be67bb340598f2d0825368101af"
format_stamp: "Formatted at 2026-09-28 10:49:00 on dist-test-slave-mn38"
server_key: "dcfa0195345df9f3ea52b614ce216eff"
server_key_iv: "977810bc9d0d9b60ae691e3618a40db4"
server_key_version: "encryptionkey@0"
I20260928 10:49:00.685665 26972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:00.705219 26972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:00.706238 26972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:00.706569 26972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:00.707719 26972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:00.709291 26972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:00.709399 26972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:00.709529 26972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:00.709596 26972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:00.741179 26972 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:46059
I20260928 10:49:00.741222 27098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:46059 every 8 connection(s)
I20260928 10:49:00.742501 26972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:49:00.747134 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 26972
I20260928 10:49:00.747298 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:49:00.747709 9067 external_mini_cluster.cc:1428] Setting key f6d02bbf1e77d3d9c0789c3ee40b44d5
I20260928 10:49:00.750823 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45079
--builtin_ntp_servers=127.8.218.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:49:00.756412 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:00.745194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:51723 (local address 127.8.218.254:45079)
0928 10:49:00.745427 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:00.745454 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:00.746954 (+ 1500us) server_negotiation.cc:407] Connection header received
0928 10:49:00.748858 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:00.748862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:00.748932 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:00.749019 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:00.750342 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:00.751387 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:00.753316 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:00.753888 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:00.754014 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:00.755581 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:00.755588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:00.755593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:00.755622 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:00.755665 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:00.755682 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:00.755776 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:00.755980 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 10:49:00.756099 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 10:49:00.757467 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:00.745779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45079 (local address 127.8.218.194:51723)
0928 10:49:00.746770 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:00.746817 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:00.748543 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:00.749332 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:00.749346 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:00.749510 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:00.750157 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:00.750176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:00.751555 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:00.751559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:00.753138 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:00.753153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:00.754140 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:00.754146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:00.754391 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:00.755181 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:00.755185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:00.755242 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:00.755348 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:00.755781 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:00.755788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:00.755794 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 10:49:00.755956 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 10:49:00.756316 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":301,"threads_started":1}
I20260928 10:49:00.759092 27099 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45079
I20260928 10:49:00.759444 27099 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:00.760386 27099 heartbeater.cc:511] Master 127.8.218.254:45079 requested a full tablet report, sending...
I20260928 10:49:00.762094 26782 ts_manager.cc:194] Registered new tserver with Master: 45690be67bb340598f2d0825368101af (127.8.218.194:46059)
I20260928 10:49:00.762907 26782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.194:51723
W20260928 10:49:01.092480 27102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:01.092808 27102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:01.092921 27102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:01.100425 27102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:01.100541 27102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:01.100680 27102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:49:01.110291 27102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:45079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:01.111677 27102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:01.113508 27102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:01.128714 27107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:01.128575 27108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:01.128551 27110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:01.129549 27102 server_base.cc:1034] running on GCE node
I20260928 10:49:01.130236 27102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:01.131316 27102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:01.133095 27102 hybrid_clock.cc:648] HybridClock initialized: now 1790592541133032 us; error 40 us; skew 500 ppm
I20260928 10:49:01.133486 27102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:01.135968 27102 webserver.cc:460] Webserver started at http://127.8.218.195:34115/ using document root <none> and password file <none>
I20260928 10:49:01.136718 27102 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:01.136827 27102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:01.137148 27102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:01.140285 27102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "359009c9543d450989f71914ed59b6cc"
format_stamp: "Formatted at 2026-09-28 10:49:01 on dist-test-slave-mn38"
server_key: "12b800ea18f7e969768f012fde2af6de"
server_key_iv: "a6b9b473364a06e10904bcd7f9a60e26"
server_key_version: "encryptionkey@0"
I20260928 10:49:01.141170 27102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "359009c9543d450989f71914ed59b6cc"
format_stamp: "Formatted at 2026-09-28 10:49:01 on dist-test-slave-mn38"
server_key: "12b800ea18f7e969768f012fde2af6de"
server_key_iv: "a6b9b473364a06e10904bcd7f9a60e26"
server_key_version: "encryptionkey@0"
I20260928 10:49:01.147346 27102 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 10:49:01.151767 27116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:01.153389 27102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:49:01.153658 27102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "359009c9543d450989f71914ed59b6cc"
format_stamp: "Formatted at 2026-09-28 10:49:01 on dist-test-slave-mn38"
server_key: "12b800ea18f7e969768f012fde2af6de"
server_key_iv: "a6b9b473364a06e10904bcd7f9a60e26"
server_key_version: "encryptionkey@0"
I20260928 10:49:01.153937 27102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:01.170897 27102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:01.171830 27102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:01.172128 27102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:01.173171 27102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:01.174963 27102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:01.175078 27102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:01.175175 27102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:01.175231 27102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:01.208043 27102 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:42427
I20260928 10:49:01.208149 27228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:42427 every 8 connection(s)
I20260928 10:49:01.209576 27102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:49:01.217578 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 27102
I20260928 10:49:01.217772 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:49:01.218279 9067 external_mini_cluster.cc:1428] Setting key 38922ac032ddc3435ca52b05f400dcf4
I20260928 10:49:01.224524 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.212443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:50697 (local address 127.8.218.254:45079)
0928 10:49:01.212658 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.212662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.214304 (+ 1642us) server_negotiation.cc:407] Connection header received
0928 10:49:01.216295 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.216301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.216408 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.216502 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:01.218115 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.219143 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.221466 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.222045 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.222169 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.223705 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:01.223711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:01.223715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:01.223737 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:01.223775 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:01.223787 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:01.223863 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.224151 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.224252 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:49:01.225754 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.213094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:45079 (local address 127.8.218.195:50697)
0928 10:49:01.214121 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.214161 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:01.215945 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:01.216801 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:01.216810 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:01.216917 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:01.217834 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.217866 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.219355 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.219360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.221244 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.221263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.222302 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.222308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.222557 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.223313 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:01.223316 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:01.223370 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:01.223472 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:01.223886 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:01.223895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:01.223902 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:49:01.224080 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 10:49:01.224440 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":355,"threads_started":1}
I20260928 10:49:01.227392 27229 heartbeater.cc:347] Connected to a master server at 127.8.218.254:45079
I20260928 10:49:01.227770 27229 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:01.228592 27229 heartbeater.cc:511] Master 127.8.218.254:45079 requested a full tablet report, sending...
I20260928 10:49:01.230239 26782 ts_manager.cc:194] Registered new tserver with Master: 359009c9543d450989f71914ed59b6cc (127.8.218.195:42427)
I20260928 10:49:01.231144 26782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.195:50697
I20260928 10:49:01.234730 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:49:01.244766 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.238951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.8.218.254:45079)
0928 10:49:01.239142 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.239147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.239534 (+ 387us) server_negotiation.cc:407] Connection header received
0928 10:49:01.239689 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.239693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.239760 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.239849 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:01.240310 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.241042 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.242471 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.242902 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.242999 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.244122 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:01.244128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:01.244131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:01.244151 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:01.244188 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:01.244201 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:01.244281 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.244424 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.244521 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:49:01.249006 26782 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:49:01.256248 26782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:49:01.282822 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.271273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:42427 (local address 127.0.0.1:47070)
0928 10:49:01.272180 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.272199 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:01.272319 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:01.273704 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:01.273714 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:01.273753 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:01.274229 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.274242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.276670 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.276676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.279385 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.279410 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.281711 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.281720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.282011 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.282080 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 10:49:01.282205 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 10:49:01.282430 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":1604,"threads_started":1}
I20260928 10:49:01.285156 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.271912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.8.218.195:42427)
0928 10:49:01.272922 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.272932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.272968 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:49:01.273089 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.273097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.273392 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.274468 (+ 1076us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:01.274758 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.276487 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.279712 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.281401 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.283683 (+ 2282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.284073 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.284159 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.284361 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":746,"threads_started":1}
I20260928 10:49:01.289849 27164 tablet_service.cc:1467] Processing CreateTablet for tablet bbf567fedd69426190565f66255d46a2 (DEFAULT_TABLE table=test-table [id=d06f58db24d74485b69547222dc6070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:01.291471 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf567fedd69426190565f66255d46a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:01.291698 26968 heartbeater.cc:503] Master 127.8.218.254:45079 was elected leader, sending a full tablet report...
I20260928 10:49:01.293463 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.270347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:46059 (local address 127.0.0.1:36922)
0928 10:49:01.271238 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.271273 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:01.271466 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:01.286822 (+ 15356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:01.286827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:01.286856 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:01.287256 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.287270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.288772 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.288778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.290455 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.290471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.292501 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.292511 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.292718 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.292754 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 10:49:01.292855 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 10:49:01.293028 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":1,"thread_start_us":424,"threads_started":1}
I20260928 10:49:01.293800 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.271653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41663 (local address 127.0.0.1:58926)
0928 10:49:01.283337 (+ 11684us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.283359 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:01.283484 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:01.285505 (+ 2021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:01.285512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:01.285542 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:01.285672 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.285679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.288044 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.288050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.289988 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.290015 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.292520 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.292527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.293191 (+ 664us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.293230 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 10:49:01.293314 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 10:49:01.293491 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11539,"mutex_wait_us":26,"thread_start_us":13199,"threads_started":1}
I20260928 10:49:01.294401 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.270558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.8.218.194:46059)
0928 10:49:01.285108 (+ 14550us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.285116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.285144 (+ 28us) server_negotiation.cc:407] Connection header received
0928 10:49:01.285238 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.285246 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.286604 (+ 1358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.286830 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:01.287446 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.288637 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.290744 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.292205 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.292393 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.292726 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.293916 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.294108 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14377,"thread_start_us":14470,"threads_started":1}
I20260928 10:49:01.294938 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.272469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.8.218.193:41663)
0928 10:49:01.284753 (+ 12284us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.284763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.284799 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:49:01.284915 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.284923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.285227 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.285510 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:01.285773 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.287858 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.290194 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.291873 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.292071 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.292381 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.294258 (+ 1877us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.294438 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12094,"thread_start_us":11950,"threads_started":1}
I20260928 10:49:01.298911 26903 tablet_service.cc:1467] Processing CreateTablet for tablet bbf567fedd69426190565f66255d46a2 (DEFAULT_TABLE table=test-table [id=d06f58db24d74485b69547222dc6070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:01.298911 27034 tablet_service.cc:1467] Processing CreateTablet for tablet bbf567fedd69426190565f66255d46a2 (DEFAULT_TABLE table=test-table [id=d06f58db24d74485b69547222dc6070d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:01.300586 27034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf567fedd69426190565f66255d46a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:01.301353 26903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf567fedd69426190565f66255d46a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:01.304809 27247 tablet_bootstrap.cc:492] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Bootstrap starting.
I20260928 10:49:01.307974 27247 tablet_bootstrap.cc:654] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:01.308990 27247 log.cc:824] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:01.312206 27247 tablet_bootstrap.cc:492] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: No bootstrap required, opened a new log
I20260928 10:49:01.312500 27247 ts_tablet_manager.cc:1397] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 10:49:01.315773 27249 tablet_bootstrap.cc:492] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Bootstrap starting.
I20260928 10:49:01.316773 27250 tablet_bootstrap.cc:492] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: Bootstrap starting.
I20260928 10:49:01.319801 27249 tablet_bootstrap.cc:654] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:01.319906 27250 tablet_bootstrap.cc:654] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:01.321308 27250 log.cc:824] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:01.321388 27249 log.cc:824] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:01.322273 27247 raft_consensus.cc:348] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.322679 27247 raft_consensus.cc:374] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:01.322800 27247 raft_consensus.cc:729] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359009c9543d450989f71914ed59b6cc, State: Initialized, Role: FOLLOWER
I20260928 10:49:01.323581 27247 consensus_queue.cc:260] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.324906 27229 heartbeater.cc:503] Master 127.8.218.254:45079 was elected leader, sending a full tablet report...
I20260928 10:49:01.325248 27250 tablet_bootstrap.cc:492] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: No bootstrap required, opened a new log
I20260928 10:49:01.325744 27247 ts_tablet_manager.cc:1428] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260928 10:49:01.325676 27250 ts_tablet_manager.cc:1397] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 10:49:01.325843 27249 tablet_bootstrap.cc:492] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: No bootstrap required, opened a new log
I20260928 10:49:01.326210 27249 ts_tablet_manager.cc:1397] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 10:49:01.336687 27249 raft_consensus.cc:348] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.336687 27250 raft_consensus.cc:348] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.337218 27250 raft_consensus.cc:374] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:01.337216 27249 raft_consensus.cc:374] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:01.337334 27250 raft_consensus.cc:729] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601ea6e87afb4b699988f9044a821a33, State: Initialized, Role: FOLLOWER
I20260928 10:49:01.337327 27249 raft_consensus.cc:729] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45690be67bb340598f2d0825368101af, State: Initialized, Role: FOLLOWER
I20260928 10:49:01.338037 27250 consensus_queue.cc:260] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.338032 27249 consensus_queue.cc:260] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.339701 27099 heartbeater.cc:503] Master 127.8.218.254:45079 was elected leader, sending a full tablet report...
I20260928 10:49:01.340090 27250 ts_tablet_manager.cc:1428] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260928 10:49:01.340238 27249 ts_tablet_manager.cc:1428] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260928 10:49:01.461341 27230 tablet.cc:2369] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:49:01.494686 27100 tablet.cc:2369] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:01.507628 27255 raft_consensus.cc:482] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:49:01.507977 27255 raft_consensus.cc:504] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.509821 27255 leader_election.cc:288] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45690be67bb340598f2d0825368101af (127.8.218.194:46059), 359009c9543d450989f71914ed59b6cc (127.8.218.195:42427)
I20260928 10:49:01.516625 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.510405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:60975 (local address 127.8.218.195:42427)
0928 10:49:01.510678 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.510685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.511428 (+ 743us) server_negotiation.cc:407] Connection header received
0928 10:49:01.511588 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.511593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.511655 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.511734 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:01.512400 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.513125 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.515013 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.515789 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.515953 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.516026 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.516268 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.516394 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:49:01.516741 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.510367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:40789 (local address 127.8.218.194:46059)
0928 10:49:01.510636 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.510644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.511351 (+ 707us) server_negotiation.cc:407] Connection header received
0928 10:49:01.511547 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.511551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.511621 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.511694 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:01.512326 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.513037 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.515099 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.515873 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.516039 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.516108 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.516361 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.516485 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 10:49:01.516801 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.510267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:42427 (local address 127.8.218.193:60975)
0928 10:49:01.511318 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.511332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:01.511456 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:01.511757 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:01.511761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:01.511783 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:01.512235 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.512245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.513261 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.513267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.514818 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.514834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.515962 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.515968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.516120 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.516145 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:49:01.516223 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 10:49:01.516358 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":18,"thread_start_us":585,"threads_started":1}
I20260928 10:49:01.516885 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.510085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:46059 (local address 127.8.218.193:40789)
0928 10:49:01.511241 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.511258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:01.511405 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:01.511727 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:01.511731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:01.511766 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:01.512168 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.512181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.513161 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.513166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.514852 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:01.514865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.516064 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.516070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:01.516215 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.516237 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 10:49:01.516313 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 10:49:01.516437 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":76,"thread_start_us":749,"threads_started":1}
I20260928 10:49:01.518146 27054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf567fedd69426190565f66255d46a2" candidate_uuid: "601ea6e87afb4b699988f9044a821a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45690be67bb340598f2d0825368101af" is_pre_election: true
I20260928 10:49:01.518146 27184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf567fedd69426190565f66255d46a2" candidate_uuid: "601ea6e87afb4b699988f9044a821a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359009c9543d450989f71914ed59b6cc" is_pre_election: true
I20260928 10:49:01.518914 27054 raft_consensus.cc:2454] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601ea6e87afb4b699988f9044a821a33 in term 0.
I20260928 10:49:01.518910 27184 raft_consensus.cc:2454] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 601ea6e87afb4b699988f9044a821a33 in term 0.
W20260928 10:49:01.519236 26969 tablet.cc:2369] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:01.519749 26858 leader_election.cc:302] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45690be67bb340598f2d0825368101af, 601ea6e87afb4b699988f9044a821a33; no voters:
I20260928 10:49:01.520180 27255 raft_consensus.cc:2781] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:49:01.520354 27255 raft_consensus.cc:482] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:49:01.520457 27255 raft_consensus.cc:3037] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:01.522802 27255 raft_consensus.cc:504] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.523490 27255 leader_election.cc:288] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [CANDIDATE]: Term 1 election: Requested vote from peers 45690be67bb340598f2d0825368101af (127.8.218.194:46059), 359009c9543d450989f71914ed59b6cc (127.8.218.195:42427)
I20260928 10:49:01.524118 27054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf567fedd69426190565f66255d46a2" candidate_uuid: "601ea6e87afb4b699988f9044a821a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45690be67bb340598f2d0825368101af"
I20260928 10:49:01.524178 27184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf567fedd69426190565f66255d46a2" candidate_uuid: "601ea6e87afb4b699988f9044a821a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359009c9543d450989f71914ed59b6cc"
I20260928 10:49:01.524338 27054 raft_consensus.cc:3037] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:01.524372 27184 raft_consensus.cc:3037] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:01.526800 27184 raft_consensus.cc:2454] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601ea6e87afb4b699988f9044a821a33 in term 1.
I20260928 10:49:01.526825 27054 raft_consensus.cc:2454] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 601ea6e87afb4b699988f9044a821a33 in term 1.
I20260928 10:49:01.527354 26856 leader_election.cc:302] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359009c9543d450989f71914ed59b6cc, 601ea6e87afb4b699988f9044a821a33; no voters:
I20260928 10:49:01.527731 27255 raft_consensus.cc:2781] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:01.528255 27255 raft_consensus.cc:686] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 1 LEADER]: Becoming Leader. State: Replica: 601ea6e87afb4b699988f9044a821a33, State: Running, Role: LEADER
I20260928 10:49:01.528805 27255 consensus_queue.cc:237] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } }
I20260928 10:49:01.534251 26782 catalog_manager.cc:5495] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 601ea6e87afb4b699988f9044a821a33 (127.8.218.193). New cstate: current_term: 1 leader_uuid: "601ea6e87afb4b699988f9044a821a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "601ea6e87afb4b699988f9044a821a33" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 } health_report { overall_health: UNKNOWN } } }
I20260928 10:49:01.582222 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.574830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.8.218.193:41663)
0928 10:49:01.575055 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.575061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.575081 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:49:01.575428 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.575434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.575561 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.575849 (+ 288us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:01.576298 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.577260 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.579040 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.579529 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.579668 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.580393 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:01.581039 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:01.581626 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.581886 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.581998 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:49:01.593175 27054 raft_consensus.cc:1261] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 1 FOLLOWER]: Refusing update from remote peer 601ea6e87afb4b699988f9044a821a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:01.593326 27184 raft_consensus.cc:1261] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 1 FOLLOWER]: Refusing update from remote peer 601ea6e87afb4b699988f9044a821a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:01.594519 27258 consensus_queue.cc:1035] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359009c9543d450989f71914ed59b6cc" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:01.594939 27255 consensus_queue.cc:1035] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45690be67bb340598f2d0825368101af" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:01.610962 27261 mvcc.cc:205] Tried to move back new op lower bound from 7334267050352545792 to 7334267050106982400. Current Snapshot: MvccSnapshot[applied={T|T < 7334267050352545792}]
I20260928 10:49:01.633917 26782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:01.634338 26782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:01.639473 26782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51511aa8a7e340bab1612eb4b5d796f4: Sending DeleteTablet for 3 replicas of tablet bbf567fedd69426190565f66255d46a2
I20260928 10:49:01.641075 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet bbf567fedd69426190565f66255d46a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:01 UTC) from {username='slave'} at 127.0.0.1:47070
I20260928 10:49:01.641062 27034 tablet_service.cc:1514] Processing DeleteTablet for tablet bbf567fedd69426190565f66255d46a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:01 UTC) from {username='slave'} at 127.0.0.1:36922
I20260928 10:49:01.641309 26903 tablet_service.cc:1514] Processing DeleteTablet for tablet bbf567fedd69426190565f66255d46a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:01 UTC) from {username='slave'} at 127.0.0.1:58926
I20260928 10:49:01.643901 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 26841
I20260928 10:49:01.653982 27272 tablet_replica.cc:330] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: stopping tablet replica
I20260928 10:49:01.654614 27272 raft_consensus.cc:2229] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:01.655300 27272 raft_consensus.cc:2258] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:01.656240 27271 tablet_replica.cc:330] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: stopping tablet replica
I20260928 10:49:01.656651 27271 raft_consensus.cc:2229] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:01.657130 27271 raft_consensus.cc:2258] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:01.659556 27271 ts_tablet_manager.cc:1905] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:01.664392 27272 ts_tablet_manager.cc:1905] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:01.667140 27271 ts_tablet_manager.cc:1918] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:01.667369 27271 log.cc:1196] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/bbf567fedd69426190565f66255d46a2
I20260928 10:49:01.667909 27271 ts_tablet_manager.cc:1939] T bbf567fedd69426190565f66255d46a2 P 45690be67bb340598f2d0825368101af: Deleting consensus metadata
I20260928 10:49:01.670043 26768 catalog_manager.cc:4841] TS 45690be67bb340598f2d0825368101af (127.8.218.194:46059): tablet bbf567fedd69426190565f66255d46a2 (table test-table [id=d06f58db24d74485b69547222dc6070d]) successfully deleted
I20260928 10:49:01.675782 27272 ts_tablet_manager.cc:1918] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:01.677207 27273 tablet_replica.cc:330] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: stopping tablet replica
I20260928 10:49:01.677753 27273 raft_consensus.cc:2229] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:49:01.678548 27273 raft_consensus.cc:2258] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:01.680919 27273 ts_tablet_manager.cc:1905] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:01.681164 27272 log.cc:1196] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/bbf567fedd69426190565f66255d46a2
I20260928 10:49:01.681705 27272 ts_tablet_manager.cc:1939] T bbf567fedd69426190565f66255d46a2 P 359009c9543d450989f71914ed59b6cc: Deleting consensus metadata
I20260928 10:49:01.684599 26766 catalog_manager.cc:4841] TS 359009c9543d450989f71914ed59b6cc (127.8.218.195:42427): tablet bbf567fedd69426190565f66255d46a2 (table test-table [id=d06f58db24d74485b69547222dc6070d]) successfully deleted
I20260928 10:49:01.686782 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.645406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.8.218.193:41663)
0928 10:49:01.645754 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.645761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.676470 (+ 30709us) server_negotiation.cc:407] Connection header received
0928 10:49:01.676697 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.676706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.676836 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.676971 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:01.677789 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.679021 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.680939 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.681722 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.681824 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.684849 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:01.684861 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:01.684878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:01.685764 (+ 886us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:01.685831 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:01.685863 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:01.685946 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.686280 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.686391 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 10:49:01.687601 26963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:01.788009 27273 ts_tablet_manager.cc:1918] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:01.788277 27273 log.cc:1196] T bbf567fedd69426190565f66255d46a2 P 601ea6e87afb4b699988f9044a821a33: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/bbf567fedd69426190565f66255d46a2
I20260928 10:49:01.788800 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 26841
I20260928 10:49:01.806146 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 26972
W20260928 10:49:01.806151 26768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.218.193:41663 (error 108)
I20260928 10:49:01.808400 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.807624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41663 (local address 127.0.0.1:58962)
0928 10:49:01.808056 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.808177 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":230,"threads_started":1}
W20260928 10:49:01.808843 26768 catalog_manager.cc:4584] TS 601ea6e87afb4b699988f9044a821a33 (127.8.218.193:41663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbf567fedd69426190565f66255d46a2: Network error: Client connection negotiation failed: client connection to 127.8.218.193:41663: connect: Connection refused (error 111)
I20260928 10:49:01.814352 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.807131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.8.218.194:46059)
0928 10:49:01.807423 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.807429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.807454 (+ 25us) server_negotiation.cc:407] Connection header received
0928 10:49:01.807521 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.807526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.807631 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.807723 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:01.808256 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.809308 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.810864 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.811333 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.811448 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.812731 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:01.812741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:01.812757 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:01.813382 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:01.813583 (+ 201us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:01.813611 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:01.813712 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.813932 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.814058 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260928 10:49:01.815138 27094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:01.862788 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.862285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41663 (local address 127.0.0.1:58970)
0928 10:49:01.862516 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.862620 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 10:49:01.863897 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.863522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41663 (local address 127.0.0.1:58980)
0928 10:49:01.863685 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.863766 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 10:49:01.904763 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 26972
I20260928 10:49:01.919711 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 27102
I20260928 10:49:01.924706 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.924200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41663 (local address 127.0.0.1:58988)
0928 10:49:01.924400 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.924496 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 10:49:01.925999 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.925482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41663 (local address 127.0.0.1:59002)
0928 10:49:01.925695 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:01.925810 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 10:49:01.926906 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:01.920505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.8.218.195:42427)
0928 10:49:01.920695 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:01.920701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:01.920720 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:49:01.920890 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:01.920895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:01.920991 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:01.921094 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:01.921516 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.922226 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.923563 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:01.923981 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:01.924094 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:01.925294 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:01.925304 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:01.925319 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:01.926162 (+ 843us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:01.926220 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:01.926243 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:01.926354 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:01.926492 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 10:49:01.926605 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:49:01.927632 27224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:02.006651 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:02.006131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41663 (local address 127.0.0.1:59008)
0928 10:49:02.006376 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:02.006480 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 10:49:02.008031 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:02.007512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41663 (local address 127.0.0.1:59010)
0928 10:49:02.007766 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:02.007857 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:41663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 10:49:02.020512 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 27102
I20260928 10:49:02.034691 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 26751
I20260928 10:49:02.035902 26812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:02.150681 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 26751
2026-09-28T10:49:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3015 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:02 dist-test-slave-mn38 krb5kdc[27283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:02 dist-test-slave-mn38 krb5kdc[27283](info): set up 2 sockets
Sep 28 10:49:02 dist-test-slave-mn38 krb5kdc[27283](info): commencing operation
krb5kdc: starting...
W20260928 10:49:03.210428 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:03 dist-test-slave-mn38 krb5kdc[27283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:03.457000 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36453
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:36453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:03.807827 27296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:03.808167 27296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:03.808306 27296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:03.815840 27296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:03.815987 27296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:03.816057 27296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:03.816134 27296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:03.816191 27296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:03.826522 27296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:36453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:03.827940 27296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:03.829317 27296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:03.842211 27304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:03.842267 27301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:03.842243 27302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:03.843097 27296 server_base.cc:1034] running on GCE node
I20260928 10:49:03.843988 27296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:03.845731 27296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:03.846964 27296 hybrid_clock.cc:648] HybridClock initialized: now 1790592543846908 us; error 58 us; skew 500 ppm
I20260928 10:49:03.847389 27296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:03 dist-test-slave-mn38 krb5kdc[27283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:03.850997 27296 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:03.852866 27296 webserver.cc:460] Webserver started at http://127.8.218.254:34053/ using document root <none> and password file <none>
I20260928 10:49:03.853726 27296 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:03.853875 27296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:03.854205 27296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:03.858044 27296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "dc3ab538786f43cebd8077f2880016ee"
format_stamp: "Formatted at 2026-09-28 10:49:03 on dist-test-slave-mn38"
server_key: "b29068170fe7ae45b655d31b9d64f381"
server_key_iv: "21374cc00ee5bba8afca35c2238aded9"
server_key_version: "encryptionkey@0"
I20260928 10:49:03.858968 27296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "dc3ab538786f43cebd8077f2880016ee"
format_stamp: "Formatted at 2026-09-28 10:49:03 on dist-test-slave-mn38"
server_key: "b29068170fe7ae45b655d31b9d64f381"
server_key_iv: "21374cc00ee5bba8afca35c2238aded9"
server_key_version: "encryptionkey@0"
I20260928 10:49:03.865420 27296 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:49:03.870059 27311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:03.871811 27296 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 10:49:03.872053 27296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "dc3ab538786f43cebd8077f2880016ee"
format_stamp: "Formatted at 2026-09-28 10:49:03 on dist-test-slave-mn38"
server_key: "b29068170fe7ae45b655d31b9d64f381"
server_key_iv: "21374cc00ee5bba8afca35c2238aded9"
server_key_version: "encryptionkey@0"
I20260928 10:49:03.872308 27296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:03.898020 27296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:03.903121 27296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:03.903409 27296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:03.921380 27296 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:36453
I20260928 10:49:03.921339 27362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:36453 every 8 connection(s)
I20260928 10:49:03.922821 27296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:03.927027 27363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:03.931691 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 27296
I20260928 10:49:03.931847 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:03.932358 9067 external_mini_cluster.cc:1428] Setting key 98ba423d25cd846f9c7ff931b74ed9ab
I20260928 10:49:03.940076 27363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee: Bootstrap starting.
I20260928 10:49:03.944041 27363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:03.945264 27363 log.cc:824] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:49:03 dist-test-slave-mn38 krb5kdc[27283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:03.951067 27363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee: No bootstrap required, opened a new log
I20260928 10:49:03.958669 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:03.934293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.8.218.254:36453)
0928 10:49:03.935144 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:03.935165 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:03.935225 (+ 60us) server_negotiation.cc:407] Connection header received
0928 10:49:03.938064 (+ 2839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:03.938105 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:03.938748 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:03.939261 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:03.940708 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:03.942801 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:03.944346 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:03.945135 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:03.945367 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:03.949987 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:03.949997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:03.950015 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:03.950056 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:03.953626 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:03.954657 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:03.954664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:03.954673 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:03.954884 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:03.955306 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:03.955309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:03.955311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:03.955934 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:03.956190 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:03.956769 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 10:49:03.957373 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":357,"threads_started":1}
I20260928 10:49:03.960659 27363 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3ab538786f43cebd8077f2880016ee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36453 } }
I20260928 10:49:03.961164 27363 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:03.961279 27363 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3ab538786f43cebd8077f2880016ee, State: Initialized, Role: FOLLOWER
I20260928 10:49:03.961952 27363 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3ab538786f43cebd8077f2880016ee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36453 } }
I20260928 10:49:03.962253 27363 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:03.962401 27363 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:03.962544 27363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:03.965152 27363 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3ab538786f43cebd8077f2880016ee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36453 } }
I20260928 10:49:03.965683 27363 leader_election.cc:302] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc3ab538786f43cebd8077f2880016ee; no voters:
I20260928 10:49:03.966230 27363 leader_election.cc:288] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:03.966368 27368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:03.967012 27368 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [term 1 LEADER]: Becoming Leader. State: Replica: dc3ab538786f43cebd8077f2880016ee, State: Running, Role: LEADER
I20260928 10:49:03.967546 27368 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3ab538786f43cebd8077f2880016ee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36453 } }
I20260928 10:49:03.968182 27363 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:03.970402 27369 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc3ab538786f43cebd8077f2880016ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3ab538786f43cebd8077f2880016ee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36453 } } }
I20260928 10:49:03.970799 27369 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:03.971489 27370 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc3ab538786f43cebd8077f2880016ee. Latest consensus state: current_term: 1 leader_uuid: "dc3ab538786f43cebd8077f2880016ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3ab538786f43cebd8077f2880016ee" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36453 } } }
I20260928 10:49:03.973594 27370 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:03.975014 27375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:03.980255 27375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:03.990129 27375 catalog_manager.cc:1347] Generated new cluster ID: 1e35630c4a574e56be9c53e50ea804a9
I20260928 10:49:03.990249 27375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:04.013494 27375 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:04.015105 27375 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:04.024664 27375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee: Generated new TSK 0
I20260928 10:49:04.025679 27375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:49:04.086790 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36453
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:04.443416 27389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:04.443711 27389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:04.443833 27389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:04.451457 27389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:04.451594 27389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:04.451752 27389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:49:04.462663 27389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:04.464003 27389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:04.465384 27389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:04.479342 27397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:04.479406 27394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:04.479424 27395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:04.479902 27389 server_base.cc:1034] running on GCE node
I20260928 10:49:04.480721 27389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:04.481838 27389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:04.483153 27389 hybrid_clock.cc:648] HybridClock initialized: now 1790592544483088 us; error 49 us; skew 500 ppm
I20260928 10:49:04.483572 27389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:04 dist-test-slave-mn38 krb5kdc[27283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592544, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:04.487178 27389 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:49:04.488787 27389 webserver.cc:460] Webserver started at http://127.8.218.193:43015/ using document root <none> and password file <none>
I20260928 10:49:04.489607 27389 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:04.489744 27389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:04.490095 27389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:04.493345 27389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba"
format_stamp: "Formatted at 2026-09-28 10:49:04 on dist-test-slave-mn38"
server_key: "ff818a709d42fada5afa6004f1397bfc"
server_key_iv: "292d7f61c00e4f2a01e1383d818aa5ca"
server_key_version: "encryptionkey@0"
I20260928 10:49:04.494248 27389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba"
format_stamp: "Formatted at 2026-09-28 10:49:04 on dist-test-slave-mn38"
server_key: "ff818a709d42fada5afa6004f1397bfc"
server_key_iv: "292d7f61c00e4f2a01e1383d818aa5ca"
server_key_version: "encryptionkey@0"
I20260928 10:49:04.500320 27389 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:49:04.504637 27404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:04.506279 27389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:49:04.506474 27389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba"
format_stamp: "Formatted at 2026-09-28 10:49:04 on dist-test-slave-mn38"
server_key: "ff818a709d42fada5afa6004f1397bfc"
server_key_iv: "292d7f61c00e4f2a01e1383d818aa5ca"
server_key_version: "encryptionkey@0"
I20260928 10:49:04.506690 27389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:04.543717 27389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:04.548501 27389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:04.548813 27389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:04.549955 27389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:04.551872 27389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:04.551983 27389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:04.552129 27389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:04.552183 27389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:04.584921 27389 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:43653
I20260928 10:49:04.584972 27516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:43653 every 8 connection(s)
I20260928 10:49:04.586308 27389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:49:04.592417 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 27389
I20260928 10:49:04.592613 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:49:04.593176 9067 external_mini_cluster.cc:1428] Setting key d5aba05ab768d0f070d04a2edb1351d6
Sep 28 10:49:04 dist-test-slave-mn38 krb5kdc[27283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592544, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:04.607534 27519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:04.588894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:41043 (local address 127.8.218.254:36453)
0928 10:49:04.589326 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:04.589331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:04.590218 (+ 887us) server_negotiation.cc:407] Connection header received
0928 10:49:04.592375 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:04.592381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:04.592469 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:04.592575 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:04.594630 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:04.595388 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:04.597049 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:04.597542 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:04.597662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:04.601424 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:04.601430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:04.601461 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:04.601534 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:04.604179 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:04.605037 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:04.605042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:04.605044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:04.605180 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:04.605681 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:04.605685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:04.605687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:04.605962 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:04.606109 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:04.606897 (+ 788us) server_negotiation.cc:299] Negotiation successful
0928 10:49:04.607169 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":229,"threads_started":1}
I20260928 10:49:04.608600 27520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:04.589296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36453 (local address 127.8.218.193:41043)
0928 10:49:04.590030 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:04.590090 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:04.592014 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:04.592943 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:04.592955 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:04.593801 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:04.594437 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:04.594457 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:04.595554 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:04.595558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:04.596864 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:04.596880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:04.597786 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:04.597794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:04.598044 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:04.598718 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:04.598743 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:04.601149 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:04.604347 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:04.604358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:04.604409 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:04.604901 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:04.605292 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:04.605297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:04.605301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:04.605563 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:04.606095 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:04.606103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:04.606492 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 10:49:04.606756 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 10:49:04.607336 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":305,"threads_started":1}
I20260928 10:49:04.610374 27517 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36453
I20260928 10:49:04.610698 27517 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:04.611552 27517 heartbeater.cc:511] Master 127.8.218.254:36453 requested a full tablet report, sending...
I20260928 10:49:04.613862 27328 ts_manager.cc:194] Registered new tserver with Master: 5bf7965a216a4886b7cb6fe6bc3aaeba (127.8.218.193:43653)
I20260928 10:49:04.615342 27328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:41043
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:49:04.648223 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36453
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:04.989275 27523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:04.989619 27523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:04.989758 27523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:04.997495 27523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:04.997648 27523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:04.997812 27523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:49:05.008468 27523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:05.009912 27523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:05.011372 27523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:05.026594 27529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:05.026578 27528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:05.027076 27531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:05.027994 27523 server_base.cc:1034] running on GCE node
I20260928 10:49:05.028869 27523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:05.030265 27523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:05.031546 27523 hybrid_clock.cc:648] HybridClock initialized: now 1790592545031491 us; error 45 us; skew 500 ppm
I20260928 10:49:05.031996 27523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:05 dist-test-slave-mn38 krb5kdc[27283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592545, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:05.036566 27523 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:49:05.038357 27523 webserver.cc:460] Webserver started at http://127.8.218.194:44449/ using document root <none> and password file <none>
I20260928 10:49:05.039116 27523 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:05.039230 27523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:05.039556 27523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:05.042870 27523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "2778f2e101ad4547a07d529741f30f7c"
format_stamp: "Formatted at 2026-09-28 10:49:05 on dist-test-slave-mn38"
server_key: "d0a4ddaadd1281a0ecf05b46cfbfeed4"
server_key_iv: "1e1688d827721717805c87313b096509"
server_key_version: "encryptionkey@0"
I20260928 10:49:05.043704 27523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "2778f2e101ad4547a07d529741f30f7c"
format_stamp: "Formatted at 2026-09-28 10:49:05 on dist-test-slave-mn38"
server_key: "d0a4ddaadd1281a0ecf05b46cfbfeed4"
server_key_iv: "1e1688d827721717805c87313b096509"
server_key_version: "encryptionkey@0"
I20260928 10:49:05.049484 27523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:49:05.053468 27538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:05.055244 27523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 10:49:05.055433 27523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "2778f2e101ad4547a07d529741f30f7c"
format_stamp: "Formatted at 2026-09-28 10:49:05 on dist-test-slave-mn38"
server_key: "d0a4ddaadd1281a0ecf05b46cfbfeed4"
server_key_iv: "1e1688d827721717805c87313b096509"
server_key_version: "encryptionkey@0"
I20260928 10:49:05.055665 27523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:05.071332 27523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:05.076053 27523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:05.076313 27523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:05.077346 27523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:05.078801 27523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:05.078879 27523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:05.078994 27523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:05.079058 27523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:05.115388 27523 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:45083
I20260928 10:49:05.115399 27650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:45083 every 8 connection(s)
I20260928 10:49:05.116921 27523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:49:05.124244 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 27523
I20260928 10:49:05.124601 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:49:05.125061 9067 external_mini_cluster.cc:1428] Setting key fa8ef780f738ab8ac6da716ce595c4fe
Sep 28 10:49:05 dist-test-slave-mn38 krb5kdc[27283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592545, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
I20260928 10:49:05.156524 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.120129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:57131 (local address 127.8.218.254:36453)
0928 10:49:05.120772 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:05.120779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:05.121910 (+ 1131us) server_negotiation.cc:407] Connection header received
0928 10:49:05.125344 (+ 3434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:05.125351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:05.125649 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:05.126192 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:05.129874 (+ 3682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.136025 (+ 6151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.139560 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.140295 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.140488 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.146689 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:05.146699 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:05.146704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:05.146764 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:05.151388 (+ 4624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:05.152800 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:05.152809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:05.152813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:05.153023 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:05.153742 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:05.153748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:05.153751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:05.154257 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:05.154444 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:05.155964 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0928 10:49:05.156186 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":246,"threads_started":1}
I20260928 10:49:05.158629 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.120634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36453 (local address 127.8.218.194:57131)
0928 10:49:05.121678 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:05.121747 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:05.124797 (+ 3050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:05.127047 (+ 2250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:05.127074 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:05.128112 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:05.129527 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.129574 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.136365 (+ 6791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.136372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.139148 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.139183 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.140700 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.140710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.141162 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.142073 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:05.142110 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:05.146239 (+ 4129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:05.151711 (+ 5472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:05.151725 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:05.151750 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:05.152561 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:05.153200 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:05.153206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:05.153210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:05.153578 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:05.154496 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:05.154508 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:05.155345 (+ 837us) client_negotiation.cc:769] Sending connection context
0928 10:49:05.155703 (+ 358us) client_negotiation.cc:240] Negotiation successful
0928 10:49:05.156729 (+ 1026us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":355,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
I20260928 10:49:05.161401 27651 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36453
I20260928 10:49:05.161826 27651 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:05.162830 27651 heartbeater.cc:511] Master 127.8.218.254:36453 requested a full tablet report, sending...
I20260928 10:49:05.165313 27328 ts_manager.cc:194] Registered new tserver with Master: 2778f2e101ad4547a07d529741f30f7c (127.8.218.194:45083)
I20260928 10:49:05.167309 27328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:57131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:49:05.191056 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36453
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:05.539659 27657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:05.539984 27657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:05.540091 27657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:05.547847 27657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:05.547976 27657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:05.548141 27657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:49:05.558470 27657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:05.559836 27657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:05.561188 27657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:05.574319 27662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:05.574306 27665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:05.574255 27663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:05.574905 27657 server_base.cc:1034] running on GCE node
I20260928 10:49:05.575595 27657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:05.576607 27657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:05.577898 27657 hybrid_clock.cc:648] HybridClock initialized: now 1790592545577817 us; error 75 us; skew 500 ppm
I20260928 10:49:05.578217 27657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:05 dist-test-slave-mn38 krb5kdc[27283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592545, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:05.581282 27657 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:49:05.582836 27657 webserver.cc:460] Webserver started at http://127.8.218.195:46269/ using document root <none> and password file <none>
I20260928 10:49:05.583539 27657 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:05.583657 27657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:05.583953 27657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:05.587024 27657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "32bb589b687b4747b573af3cf0a14fea"
format_stamp: "Formatted at 2026-09-28 10:49:05 on dist-test-slave-mn38"
server_key: "5479541f7c4109fabb20cc67fb7b6e2e"
server_key_iv: "6de6985da56ce3a27be7f65407ea71d1"
server_key_version: "encryptionkey@0"
I20260928 10:49:05.587862 27657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "32bb589b687b4747b573af3cf0a14fea"
format_stamp: "Formatted at 2026-09-28 10:49:05 on dist-test-slave-mn38"
server_key: "5479541f7c4109fabb20cc67fb7b6e2e"
server_key_iv: "6de6985da56ce3a27be7f65407ea71d1"
server_key_version: "encryptionkey@0"
I20260928 10:49:05.593915 27657 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:49:05.599157 27672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:05.601063 27657 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:49:05.601294 27657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "32bb589b687b4747b573af3cf0a14fea"
format_stamp: "Formatted at 2026-09-28 10:49:05 on dist-test-slave-mn38"
server_key: "5479541f7c4109fabb20cc67fb7b6e2e"
server_key_iv: "6de6985da56ce3a27be7f65407ea71d1"
server_key_version: "encryptionkey@0"
I20260928 10:49:05.601612 27657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:05.618597 27517 heartbeater.cc:503] Master 127.8.218.254:36453 was elected leader, sending a full tablet report...
I20260928 10:49:05.628698 27657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:05.634234 27657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:05.634554 27657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:05.635699 27657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:05.637214 27657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:05.637296 27657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:05.637401 27657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:05.637491 27657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 10:49:05.670974 27657 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:37239
I20260928 10:49:05.671017 27784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:37239 every 8 connection(s)
I20260928 10:49:05.672267 27657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:49:05.679656 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 27657
I20260928 10:49:05.679841 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:49:05.680356 9067 external_mini_cluster.cc:1428] Setting key 7e537e35566b23d0910ae64dd1514404
Sep 28 10:49:05 dist-test-slave-mn38 krb5kdc[27283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592545, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:05.693704 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.674899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:36495 (local address 127.8.218.254:36453)
0928 10:49:05.675352 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:05.675357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:05.676368 (+ 1011us) server_negotiation.cc:407] Connection header received
0928 10:49:05.678346 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:05.678352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:05.678442 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:05.678546 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:05.680564 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.681600 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.683614 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.684072 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.684176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.687714 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:05.687719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:05.687723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:05.687760 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:05.690414 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:05.691213 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:05.691217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:05.691220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:05.691388 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:05.691822 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:05.691827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:05.691829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:05.692157 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:05.692299 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:05.693114 (+ 815us) server_negotiation.cc:299] Negotiation successful
0928 10:49:05.693352 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":219,"threads_started":1}
I20260928 10:49:05.694928 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.675389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:36453 (local address 127.8.218.195:36495)
0928 10:49:05.676208 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:05.676247 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:05.678060 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:05.678798 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:05.678807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:05.679436 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:05.680306 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.680336 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.681801 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.681807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.683473 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.683489 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.684282 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.684288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.684505 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.685136 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:05.685160 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:05.687458 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:05.690574 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:05.690583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:05.690598 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:05.691091 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:05.691488 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:05.691491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:05.691493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:05.691689 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:05.692312 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:05.692319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:05.692698 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 10:49:05.692969 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 10:49:05.693530 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":301,"threads_started":1}
I20260928 10:49:05.696624 27785 heartbeater.cc:347] Connected to a master server at 127.8.218.254:36453
I20260928 10:49:05.696942 27785 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:05.697749 27785 heartbeater.cc:511] Master 127.8.218.254:36453 requested a full tablet report, sending...
I20260928 10:49:05.699507 27328 ts_manager.cc:194] Registered new tserver with Master: 32bb589b687b4747b573af3cf0a14fea (127.8.218.195:37239)
I20260928 10:49:05.700526 27328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:36495
I20260928 10:49:05.709882 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:49:05.728386 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.714320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.8.218.254:36453)
0928 10:49:05.714482 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:05.714486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:05.714919 (+ 433us) server_negotiation.cc:407] Connection header received
0928 10:49:05.715086 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:05.715091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:05.715166 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:05.715262 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:05.716783 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.717600 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.719083 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.719527 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.719640 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.722529 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:05.722536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:05.722540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:05.722585 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:05.725273 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:05.726243 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:05.726248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:05.726251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:05.726408 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:05.726950 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:05.726954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:05.726957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:05.727329 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:05.727571 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:05.727933 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 10:49:05.728101 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:49:05.732780 27328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:49:05.740016 27328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:49:05.771397 27801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.755305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.8.218.193:43653)
0928 10:49:05.760038 (+ 4733us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:05.760050 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:05.760085 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:49:05.760214 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:05.760224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:05.760554 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:05.763518 (+ 2964us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:05.763851 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.765535 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.768546 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.770138 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.770255 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.770475 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:05.770860 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 10:49:05.771026 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4563,"thread_start_us":4342,"threads_started":1}
I20260928 10:49:05.771497 27799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.754719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:43653 (local address 127.0.0.1:58682)
0928 10:49:05.759268 (+ 4549us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:05.759292 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:05.759431 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:05.760889 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:05.760900 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:05.760942 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:05.761554 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.761571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.765724 (+ 4153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.765747 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.768298 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.768313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.770368 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.770375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.770623 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.770666 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 10:49:05.770788 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 10:49:05.771026 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4371,"mutex_wait_us":5,"thread_start_us":1881,"threads_started":1}
I20260928 10:49:05.775230 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.754524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:45083 (local address 127.0.0.1:59106)
0928 10:49:05.756564 (+ 2040us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:05.756625 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:05.756861 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:05.767037 (+ 10176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:05.767047 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:05.767097 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:05.767568 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.767581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.769876 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.769884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.772404 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.772414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.774342 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.774348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.774524 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.774556 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:49:05.774626 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 10:49:05.774753 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1716,"mutex_wait_us":92,"spinlock_wait_cycles":10496,"thread_start_us":2619,"threads_started":1}
I20260928 10:49:05.775475 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.757912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:37239 (local address 127.0.0.1:60478)
0928 10:49:05.765756 (+ 7844us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:05.765776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:05.765911 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:05.766956 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:05.766961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:05.766995 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:05.767492 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.767506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.769824 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.769829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.772354 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.772374 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.773981 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.773988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.774170 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.774202 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:49:05.774302 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 10:49:05.774465 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7697,"mutex_wait_us":83,"spinlock_wait_cycles":896,"thread_start_us":7539,"threads_started":1}
I20260928 10:49:05.775905 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.757965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.8.218.195:37239)
0928 10:49:05.765426 (+ 7461us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:05.766106 (+ 680us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:05.766142 (+ 36us) server_negotiation.cc:407] Connection header received
0928 10:49:05.766231 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:05.766237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:05.766478 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:05.767752 (+ 1274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:05.768197 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.769694 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.772557 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.773711 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.773848 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.774094 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:05.774958 (+ 864us) server_negotiation.cc:299] Negotiation successful
0928 10:49:05.775150 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7241,"thread_start_us":7049,"threads_started":1}
I20260928 10:49:05.776497 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.754524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.8.218.194:45083)
0928 10:49:05.766433 (+ 11909us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:05.766443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:05.766472 (+ 29us) server_negotiation.cc:407] Connection header received
0928 10:49:05.766572 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:05.766579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:05.766807 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:05.767773 (+ 966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:05.768053 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.769711 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.772588 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.774098 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.775668 (+ 1570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.775863 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:05.775942 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 10:49:05.776102 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11725,"thread_start_us":12109,"threads_started":1}
I20260928 10:49:05.776254 27452 tablet_service.cc:1467] Processing CreateTablet for tablet f2938d0dc49540e1a626b2d430efea54 (DEFAULT_TABLE table=test-table [id=4de531aefef04e8cb11c0c5f9c650dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:05.777885 27452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2938d0dc49540e1a626b2d430efea54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:05.779830 27586 tablet_service.cc:1467] Processing CreateTablet for tablet f2938d0dc49540e1a626b2d430efea54 (DEFAULT_TABLE table=test-table [id=4de531aefef04e8cb11c0c5f9c650dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:05.779721 27720 tablet_service.cc:1467] Processing CreateTablet for tablet f2938d0dc49540e1a626b2d430efea54 (DEFAULT_TABLE table=test-table [id=4de531aefef04e8cb11c0c5f9c650dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:05.781159 27720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2938d0dc49540e1a626b2d430efea54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:05.781163 27586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2938d0dc49540e1a626b2d430efea54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:05.790529 27804 tablet_bootstrap.cc:492] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Bootstrap starting.
I20260928 10:49:05.794097 27804 tablet_bootstrap.cc:654] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:05.795235 27804 log.cc:824] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:05.796506 27805 tablet_bootstrap.cc:492] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Bootstrap starting.
I20260928 10:49:05.797541 27806 tablet_bootstrap.cc:492] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Bootstrap starting.
I20260928 10:49:05.802078 27806 tablet_bootstrap.cc:654] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:05.803375 27806 log.cc:824] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:05.805321 27804 tablet_bootstrap.cc:492] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: No bootstrap required, opened a new log
I20260928 10:49:05.805626 27805 tablet_bootstrap.cc:654] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:05.805689 27804 ts_tablet_manager.cc:1397] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260928 10:49:05.806793 27805 log.cc:824] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:05.806905 27806 tablet_bootstrap.cc:492] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: No bootstrap required, opened a new log
I20260928 10:49:05.807215 27806 ts_tablet_manager.cc:1397] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260928 10:49:05.810698 27805 tablet_bootstrap.cc:492] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: No bootstrap required, opened a new log
I20260928 10:49:05.811136 27805 ts_tablet_manager.cc:1397] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20260928 10:49:05.817391 27804 raft_consensus.cc:348] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:05.818099 27804 raft_consensus.cc:374] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:05.818230 27804 raft_consensus.cc:729] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf7965a216a4886b7cb6fe6bc3aaeba, State: Initialized, Role: FOLLOWER
I20260928 10:49:05.819005 27804 consensus_queue.cc:260] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:05.818953 27806 raft_consensus.cc:348] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:05.819042 27805 raft_consensus.cc:348] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:05.819450 27806 raft_consensus.cc:374] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:05.819456 27805 raft_consensus.cc:374] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:05.819561 27805 raft_consensus.cc:729] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2778f2e101ad4547a07d529741f30f7c, State: Initialized, Role: FOLLOWER
I20260928 10:49:05.819571 27806 raft_consensus.cc:729] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bb589b687b4747b573af3cf0a14fea, State: Initialized, Role: FOLLOWER
I20260928 10:49:05.820369 27806 consensus_queue.cc:260] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:05.820372 27805 consensus_queue.cc:260] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:05.821245 27804 ts_tablet_manager.cc:1428] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260928 10:49:05.822378 27785 heartbeater.cc:503] Master 127.8.218.254:36453 was elected leader, sending a full tablet report...
I20260928 10:49:05.823109 27806 ts_tablet_manager.cc:1428] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260928 10:49:05.823601 27651 heartbeater.cc:503] Master 127.8.218.254:36453 was elected leader, sending a full tablet report...
I20260928 10:49:05.823772 27805 ts_tablet_manager.cc:1428] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
W20260928 10:49:05.838774 27518 tablet.cc:2369] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:49:05.869679 27652 tablet.cc:2369] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:49:05.923830 27786 tablet.cc:2369] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:05.985800 27811 raft_consensus.cc:482] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:49:05.986129 27811 raft_consensus.cc:504] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:05.987869 27811 leader_election.cc:288] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2778f2e101ad4547a07d529741f30f7c (127.8.218.194:45083), 5bf7965a216a4886b7cb6fe6bc3aaeba (127.8.218.193:43653)
I20260928 10:49:05.993739 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.988420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:43653 (local address 127.8.218.195:50615)
0928 10:49:05.988604 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:05.988630 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:05.988769 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:05.989128 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:05.989133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:05.989167 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:05.989331 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.989343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.990354 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.990357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.991715 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.991731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.992927 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.992933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.993136 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.993173 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 10:49:05.993274 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 10:49:05.993420 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 10:49:05.993983 27801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.988363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:50615 (local address 127.8.218.193:43653)
0928 10:49:05.988742 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:05.988747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:05.988768 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:49:05.988916 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:05.988920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:05.988991 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:05.989064 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:05.989498 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.990212 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.991873 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.992764 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.992945 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.993033 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:05.993585 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 10:49:05.993733 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260928 10:49:05.994961 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.988503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:53427 (local address 127.8.218.194:45083)
0928 10:49:05.988711 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:05.988717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:05.989472 (+ 755us) server_negotiation.cc:407] Connection header received
0928 10:49:05.989645 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:05.989649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:05.989713 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:05.989800 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:05.990309 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.990997 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.992725 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.994077 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.994232 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.994303 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:05.994570 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 10:49:05.994700 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:49:05.995007 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:05.988503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:45083 (local address 127.8.218.195:53427)
0928 10:49:05.989348 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:05.989363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:05.989512 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:05.989829 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:05.989833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:05.989857 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:05.990146 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.990155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.991144 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.991147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.992511 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:05.992523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:05.994270 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:05.994274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:05.994399 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:05.994423 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 10:49:05.994518 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 10:49:05.994653 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":433,"threads_started":1}
I20260928 10:49:05.994894 27472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2938d0dc49540e1a626b2d430efea54" candidate_uuid: "32bb589b687b4747b573af3cf0a14fea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" is_pre_election: true
I20260928 10:49:05.995514 27472 raft_consensus.cc:2454] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32bb589b687b4747b573af3cf0a14fea in term 0.
I20260928 10:49:05.996315 27606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2938d0dc49540e1a626b2d430efea54" candidate_uuid: "32bb589b687b4747b573af3cf0a14fea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2778f2e101ad4547a07d529741f30f7c" is_pre_election: true
I20260928 10:49:05.996405 27676 leader_election.cc:302] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32bb589b687b4747b573af3cf0a14fea, 5bf7965a216a4886b7cb6fe6bc3aaeba; no voters:
I20260928 10:49:05.996917 27606 raft_consensus.cc:2454] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32bb589b687b4747b573af3cf0a14fea in term 0.
I20260928 10:49:05.997085 27811 raft_consensus.cc:2781] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:49:05.997272 27811 raft_consensus.cc:482] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:49:05.997406 27811 raft_consensus.cc:3037] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:06.000080 27811 raft_consensus.cc:504] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:06.000874 27811 leader_election.cc:288] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [CANDIDATE]: Term 1 election: Requested vote from peers 2778f2e101ad4547a07d529741f30f7c (127.8.218.194:45083), 5bf7965a216a4886b7cb6fe6bc3aaeba (127.8.218.193:43653)
I20260928 10:49:06.001524 27606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2938d0dc49540e1a626b2d430efea54" candidate_uuid: "32bb589b687b4747b573af3cf0a14fea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2778f2e101ad4547a07d529741f30f7c"
I20260928 10:49:06.001682 27472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2938d0dc49540e1a626b2d430efea54" candidate_uuid: "32bb589b687b4747b573af3cf0a14fea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba"
I20260928 10:49:06.001749 27606 raft_consensus.cc:3037] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:06.001915 27472 raft_consensus.cc:3037] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:06.004173 27606 raft_consensus.cc:2454] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32bb589b687b4747b573af3cf0a14fea in term 1.
I20260928 10:49:06.004175 27472 raft_consensus.cc:2454] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32bb589b687b4747b573af3cf0a14fea in term 1.
I20260928 10:49:06.004685 27676 leader_election.cc:302] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32bb589b687b4747b573af3cf0a14fea, 5bf7965a216a4886b7cb6fe6bc3aaeba; no voters:
I20260928 10:49:06.005036 27811 raft_consensus.cc:2781] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:06.005529 27811 raft_consensus.cc:686] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 1 LEADER]: Becoming Leader. State: Replica: 32bb589b687b4747b573af3cf0a14fea, State: Running, Role: LEADER
I20260928 10:49:06.006069 27811 consensus_queue.cc:237] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } }
I20260928 10:49:06.010845 27328 catalog_manager.cc:5495] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea reported cstate change: term changed from 0 to 1, leader changed from <none> to 32bb589b687b4747b573af3cf0a14fea (127.8.218.195). New cstate: current_term: 1 leader_uuid: "32bb589b687b4747b573af3cf0a14fea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32bb589b687b4747b573af3cf0a14fea" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 37239 } health_report { overall_health: HEALTHY } } }
I20260928 10:49:06.071915 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:06.066227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.8.218.195:37239)
0928 10:49:06.066458 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:06.066463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:06.066482 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:49:06.066577 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:06.066581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:06.066659 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:06.066795 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:06.067218 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:06.068023 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:06.069499 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:06.070056 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:06.070161 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:06.070741 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:06.071274 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:06.071430 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:06.071575 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 10:49:06.071626 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 10:49:06.080497 27472 raft_consensus.cc:1261] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 1 FOLLOWER]: Refusing update from remote peer 32bb589b687b4747b573af3cf0a14fea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:06.080498 27606 raft_consensus.cc:1261] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 1 FOLLOWER]: Refusing update from remote peer 32bb589b687b4747b573af3cf0a14fea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:06.081542 27814 consensus_queue.cc:1035] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2778f2e101ad4547a07d529741f30f7c" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:06.081955 27811 consensus_queue.cc:1035] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf7965a216a4886b7cb6fe6bc3aaeba" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 43653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:06.094969 27818 mvcc.cc:205] Tried to move back new op lower bound from 7334267068735328256 to 7334267068445368320. Current Snapshot: MvccSnapshot[applied={T|T < 7334267068735328256}]
I20260928 10:49:06.116576 27326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:06.116875 27326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:06.121313 27326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc3ab538786f43cebd8077f2880016ee: Sending DeleteTablet for 3 replicas of tablet f2938d0dc49540e1a626b2d430efea54
I20260928 10:49:06.122560 27586 tablet_service.cc:1514] Processing DeleteTablet for tablet f2938d0dc49540e1a626b2d430efea54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:06 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:59106
I20260928 10:49:06.122555 27452 tablet_service.cc:1514] Processing DeleteTablet for tablet f2938d0dc49540e1a626b2d430efea54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:06 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:58682
I20260928 10:49:06.128046 27720 tablet_service.cc:1514] Processing DeleteTablet for tablet f2938d0dc49540e1a626b2d430efea54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:06 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:60478
I20260928 10:49:06.132442 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 27389
I20260928 10:49:06.139925 27828 tablet_replica.cc:330] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: stopping tablet replica
I20260928 10:49:06.140424 27828 raft_consensus.cc:2229] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:06.141692 27828 raft_consensus.cc:2258] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:06.144799 27828 ts_tablet_manager.cc:1905] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:06.146657 27829 tablet_replica.cc:330] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: stopping tablet replica
I20260928 10:49:06.147208 27829 raft_consensus.cc:2229] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:49:06.148057 27829 raft_consensus.cc:2258] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:06.151940 27827 tablet_replica.cc:330] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: stopping tablet replica
I20260928 10:49:06.152463 27827 raft_consensus.cc:2229] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:06.152788 27828 ts_tablet_manager.cc:1918] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:06.153023 27828 log.cc:1196] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/f2938d0dc49540e1a626b2d430efea54
I20260928 10:49:06.153093 27827 raft_consensus.cc:2258] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:06.153925 27828 ts_tablet_manager.cc:1939] T f2938d0dc49540e1a626b2d430efea54 P 5bf7965a216a4886b7cb6fe6bc3aaeba: Deleting consensus metadata
I20260928 10:49:06.155972 27829 ts_tablet_manager.cc:1905] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:06.156602 27315 catalog_manager.cc:4841] TS 5bf7965a216a4886b7cb6fe6bc3aaeba (127.8.218.193:43653): tablet f2938d0dc49540e1a626b2d430efea54 (table test-table [id=4de531aefef04e8cb11c0c5f9c650dd1]) successfully deleted
Sep 28 10:49:06 dist-test-slave-mn38 krb5kdc[27283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:49:06.158885 27827 ts_tablet_manager.cc:1905] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:06.164499 27829 ts_tablet_manager.cc:1918] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:06.164693 27829 log.cc:1196] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/f2938d0dc49540e1a626b2d430efea54
I20260928 10:49:06.165259 27829 ts_tablet_manager.cc:1939] T f2938d0dc49540e1a626b2d430efea54 P 32bb589b687b4747b573af3cf0a14fea: Deleting consensus metadata
I20260928 10:49:06.165794 27801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:06.133880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.8.218.193:43653)
0928 10:49:06.134081 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:06.134087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:06.145136 (+ 11049us) server_negotiation.cc:407] Connection header received
0928 10:49:06.145362 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:06.145368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:06.145489 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:06.145612 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:06.148403 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:06.149477 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:06.151434 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:06.152112 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:06.152223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:06.158378 (+ 6155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:06.158394 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:06.158411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:06.158457 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:06.162147 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:06.163121 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:06.163132 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:06.163136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:06.163308 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:06.163771 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:06.163777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:06.163780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:06.164272 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:06.164556 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:06.165145 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 10:49:06.165401 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:49:06.166631 27827 ts_tablet_manager.cc:1918] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:06.166868 27827 log.cc:1196] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/f2938d0dc49540e1a626b2d430efea54
I20260928 10:49:06.167253 27313 catalog_manager.cc:4841] TS 32bb589b687b4747b573af3cf0a14fea (127.8.218.195:37239): tablet f2938d0dc49540e1a626b2d430efea54 (table test-table [id=4de531aefef04e8cb11c0c5f9c650dd1]) successfully deleted
I20260928 10:49:06.167428 27512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:06.167557 27827 ts_tablet_manager.cc:1939] T f2938d0dc49540e1a626b2d430efea54 P 2778f2e101ad4547a07d529741f30f7c: Deleting consensus metadata
I20260928 10:49:06.169847 27313 catalog_manager.cc:4841] TS 2778f2e101ad4547a07d529741f30f7c (127.8.218.194:45083): tablet f2938d0dc49540e1a626b2d430efea54 (table test-table [id=4de531aefef04e8cb11c0c5f9c650dd1]) successfully deleted
I20260928 10:49:06.260026 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 27389
I20260928 10:49:06.275789 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 27523
Sep 28 10:49:06 dist-test-slave-mn38 krb5kdc[27283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:49:06.295631 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:06.276642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.8.218.194:45083)
0928 10:49:06.276836 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:06.276843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:06.276874 (+ 31us) server_negotiation.cc:407] Connection header received
0928 10:49:06.277085 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:06.277090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:06.277172 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:06.277258 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:06.279013 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:06.279972 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:06.282257 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:06.282686 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:06.282816 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:06.288444 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:06.288464 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:06.288482 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:06.288528 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:06.292047 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:06.293183 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:06.293192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:06.293195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:06.293398 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:06.293935 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:06.293940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:06.293942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:06.294394 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:06.294656 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:06.295040 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 10:49:06.295277 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:49:06.296414 27646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:06.386202 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 27523
I20260928 10:49:06.407717 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 27657
Sep 28 10:49:06 dist-test-slave-mn38 krb5kdc[27283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:49:06.425961 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:06.408620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.8.218.195:37239)
0928 10:49:06.408809 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:06.408820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:06.408916 (+ 96us) server_negotiation.cc:407] Connection header received
0928 10:49:06.409122 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:06.409126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:06.409209 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:06.409307 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:06.410865 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:06.411836 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:06.413207 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:06.413783 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:06.413917 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:06.418965 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:06.418983 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:06.419000 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:06.419052 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:06.422794 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:06.423633 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:06.423641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:06.423643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:06.423830 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:06.424222 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:06.424226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:06.424228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:06.424657 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:06.425044 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:06.425320 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 10:49:06.425599 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:49:06.426765 27780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:06.552961 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 27657
I20260928 10:49:06.570016 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 27296
I20260928 10:49:06.571382 27358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:06.685981 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 27296
2026-09-28T10:49:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4532 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T10:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:06Z Disabled control of system clock
I20260928 10:49:06.764925 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37497
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:37497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:49:07.111017 27838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:07.111348 27838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:07.111471 27838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:07.119186 27838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:07.119313 27838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:07.119371 27838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:07.119452 27838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:07.119506 27838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:07.129546 27838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:37497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:07.130909 27838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:07.132926 27838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:07.146358 27843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:07.146322 27844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:07.146382 27846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:07.147517 27838 server_base.cc:1034] running on GCE node
I20260928 10:49:07.148403 27838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:07.150197 27838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:07.151458 27838 hybrid_clock.cc:648] HybridClock initialized: now 1790592547151389 us; error 63 us; skew 500 ppm
I20260928 10:49:07.151885 27838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:07.154886 27838 webserver.cc:460] Webserver started at http://127.8.218.254:40345/ using document root <none> and password file <none>
I20260928 10:49:07.155710 27838 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:07.155818 27838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:07.156154 27838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:07.159451 27838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "e627a7f62d674ef3a4a7d90b47cc2484"
format_stamp: "Formatted at 2026-09-28 10:49:07 on dist-test-slave-mn38"
server_key: "8fe811d1ef7e92cf944a462301123a80"
server_key_iv: "6f729b0ec248d9a8fe107f384e4d28f0"
server_key_version: "encryptionkey@0"
I20260928 10:49:07.160375 27838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "e627a7f62d674ef3a4a7d90b47cc2484"
format_stamp: "Formatted at 2026-09-28 10:49:07 on dist-test-slave-mn38"
server_key: "8fe811d1ef7e92cf944a462301123a80"
server_key_iv: "6f729b0ec248d9a8fe107f384e4d28f0"
server_key_version: "encryptionkey@0"
I20260928 10:49:07.166162 27838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 10:49:07.170132 27852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:07.171626 27838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:07.171813 27838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "e627a7f62d674ef3a4a7d90b47cc2484"
format_stamp: "Formatted at 2026-09-28 10:49:07 on dist-test-slave-mn38"
server_key: "8fe811d1ef7e92cf944a462301123a80"
server_key_iv: "6f729b0ec248d9a8fe107f384e4d28f0"
server_key_version: "encryptionkey@0"
I20260928 10:49:07.172037 27838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:07.213482 27838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:07.214491 27838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:07.214820 27838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:07.233668 27838 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:37497
I20260928 10:49:07.233731 27903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:37497 every 8 connection(s)
I20260928 10:49:07.235077 27838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:07.239092 27904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:07.240993 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 27838
I20260928 10:49:07.241190 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:07.241773 9067 external_mini_cluster.cc:1428] Setting key a5c23bfbc554b8e5be606c092b3810aa
I20260928 10:49:07.251848 27904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484: Bootstrap starting.
I20260928 10:49:07.255262 27904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:07.256361 27904 log.cc:824] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:07.259725 27904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484: No bootstrap required, opened a new log
I20260928 10:49:07.260524 27907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:07.243821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.8.218.254:37497)
0928 10:49:07.244927 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:07.244942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:07.244991 (+ 49us) server_negotiation.cc:407] Connection header received
0928 10:49:07.246884 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:07.246910 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:07.247437 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:07.247840 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:07.248503 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:07.250056 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:07.251464 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:07.252283 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:07.252466 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:07.257885 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:07.257894 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:07.257912 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:07.258509 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:07.258596 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:07.258638 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:07.258738 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:07.259049 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 10:49:07.259401 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":391,"threads_started":1}
I20260928 10:49:07.268167 27904 raft_consensus.cc:348] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e627a7f62d674ef3a4a7d90b47cc2484" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37497 } }
I20260928 10:49:07.268558 27904 raft_consensus.cc:374] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:07.268663 27904 raft_consensus.cc:729] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e627a7f62d674ef3a4a7d90b47cc2484, State: Initialized, Role: FOLLOWER
I20260928 10:49:07.269237 27904 consensus_queue.cc:260] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e627a7f62d674ef3a4a7d90b47cc2484" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37497 } }
I20260928 10:49:07.269484 27904 raft_consensus.cc:388] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:07.269618 27904 raft_consensus.cc:482] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:07.269788 27904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:07.272065 27904 raft_consensus.cc:504] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e627a7f62d674ef3a4a7d90b47cc2484" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37497 } }
I20260928 10:49:07.272562 27904 leader_election.cc:302] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e627a7f62d674ef3a4a7d90b47cc2484; no voters:
I20260928 10:49:07.273130 27904 leader_election.cc:288] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:07.273281 27909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:07.273965 27909 raft_consensus.cc:686] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [term 1 LEADER]: Becoming Leader. State: Replica: e627a7f62d674ef3a4a7d90b47cc2484, State: Running, Role: LEADER
I20260928 10:49:07.274535 27909 consensus_queue.cc:237] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e627a7f62d674ef3a4a7d90b47cc2484" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37497 } }
I20260928 10:49:07.275142 27904 sys_catalog.cc:564] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:07.277281 27910 sys_catalog.cc:455] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e627a7f62d674ef3a4a7d90b47cc2484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e627a7f62d674ef3a4a7d90b47cc2484" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37497 } } }
I20260928 10:49:07.277685 27910 sys_catalog.cc:458] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:07.278307 27911 sys_catalog.cc:455] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e627a7f62d674ef3a4a7d90b47cc2484. Latest consensus state: current_term: 1 leader_uuid: "e627a7f62d674ef3a4a7d90b47cc2484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e627a7f62d674ef3a4a7d90b47cc2484" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37497 } } }
I20260928 10:49:07.278640 27911 sys_catalog.cc:458] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:07.279592 27915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:07.284747 27915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:07.295132 27915 catalog_manager.cc:1347] Generated new cluster ID: fbe90e19fe0747e8b78633b984b80cec
I20260928 10:49:07.295295 27915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:07.309978 27915 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:07.311542 27915 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:07.318354 27915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484: Generated new TSK 0
I20260928 10:49:07.319567 27915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:07.332279 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37497
--builtin_ntp_servers=127.8.218.212:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:49:07.706270 27928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:07.706600 27928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:07.706728 27928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:07.714495 27928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:07.714629 27928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:07.714774 27928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:49:07.724834 27928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:07.726394 27928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:07.728224 27928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:07.742561 27934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:07.742697 27933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:07.742717 27936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:07.743247 27928 server_base.cc:1034] running on GCE node
I20260928 10:49:07.744036 27928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:07.745061 27928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:07.746327 27928 hybrid_clock.cc:648] HybridClock initialized: now 1790592547746250 us; error 61 us; skew 500 ppm
I20260928 10:49:07.746731 27928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:07.749977 27928 webserver.cc:460] Webserver started at http://127.8.218.193:41099/ using document root <none> and password file <none>
I20260928 10:49:07.750882 27928 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:07.751013 27928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:07.751367 27928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:07.754896 27928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "1afdb0fbbaeb47fba02280922fe906a4"
format_stamp: "Formatted at 2026-09-28 10:49:07 on dist-test-slave-mn38"
server_key: "e2a7118089d8fec95011d007e195bd29"
server_key_iv: "6ec8bb45468a64b9151bc20e7354d03f"
server_key_version: "encryptionkey@0"
I20260928 10:49:07.755909 27928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "1afdb0fbbaeb47fba02280922fe906a4"
format_stamp: "Formatted at 2026-09-28 10:49:07 on dist-test-slave-mn38"
server_key: "e2a7118089d8fec95011d007e195bd29"
server_key_iv: "6ec8bb45468a64b9151bc20e7354d03f"
server_key_version: "encryptionkey@0"
I20260928 10:49:07.762991 27928 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 10:49:07.767030 27942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:07.768747 27928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:07.768955 27928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "1afdb0fbbaeb47fba02280922fe906a4"
format_stamp: "Formatted at 2026-09-28 10:49:07 on dist-test-slave-mn38"
server_key: "e2a7118089d8fec95011d007e195bd29"
server_key_iv: "6ec8bb45468a64b9151bc20e7354d03f"
server_key_version: "encryptionkey@0"
I20260928 10:49:07.769222 27928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:07.797740 27928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:07.798651 27928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:07.798955 27928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:07.800007 27928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:07.801754 27928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:07.801862 27928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:07.801982 27928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:07.802058 27928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:07.833599 27928 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:44929
I20260928 10:49:07.833724 28054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:44929 every 8 connection(s)
I20260928 10:49:07.834972 27928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:49:07.842703 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 27928
I20260928 10:49:07.842877 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:49:07.843418 9067 external_mini_cluster.cc:1428] Setting key c88d3baaa3f2d4e37a3bfa2dcbbf9703
I20260928 10:49:07.846608 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37497
--builtin_ntp_servers=127.8.218.212:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:49:07.849261 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:07.837677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:38085 (local address 127.8.218.254:37497)
0928 10:49:07.838131 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:07.838136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:07.839470 (+ 1334us) server_negotiation.cc:407] Connection header received
0928 10:49:07.841419 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:07.841425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:07.841557 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:07.841660 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:07.842937 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:07.844024 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:07.845981 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:07.846613 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:07.846773 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:07.848308 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:07.848315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:07.848319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:07.848345 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:07.848406 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:07.848494 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:07.848598 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:07.848817 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 10:49:07.848928 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260928 10:49:07.850306 28058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:07.838229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37497 (local address 127.8.218.193:38085)
0928 10:49:07.839273 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:07.839321 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:07.841084 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:07.841927 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:07.841938 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:07.842052 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:07.842706 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:07.842726 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:07.844206 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:07.844212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:07.845819 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:07.845835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:07.846902 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:07.846909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:07.847155 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:07.847854 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:07.847857 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:07.847902 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:07.848003 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:07.848609 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:07.848619 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:07.848626 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 10:49:07.848788 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 10:49:07.849144 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":330,"threads_started":1}
I20260928 10:49:07.852095 28055 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37497
I20260928 10:49:07.852473 28055 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:07.853479 28055 heartbeater.cc:511] Master 127.8.218.254:37497 requested a full tablet report, sending...
I20260928 10:49:07.856099 27869 ts_manager.cc:194] Registered new tserver with Master: 1afdb0fbbaeb47fba02280922fe906a4 (127.8.218.193:44929)
I20260928 10:49:07.858095 27869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.193:38085
W20260928 10:49:08.190335 28059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:08.190645 28059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:08.190737 28059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:08.198275 28059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:08.198396 28059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:08.198504 28059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:49:08.208016 28059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:08.209286 28059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:08.211042 28059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:08.224529 28064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:08.224463 28065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:08.224502 28067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:08.225473 28059 server_base.cc:1034] running on GCE node
I20260928 10:49:08.226369 28059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:08.227532 28059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:08.228757 28059 hybrid_clock.cc:648] HybridClock initialized: now 1790592548228707 us; error 50 us; skew 500 ppm
I20260928 10:49:08.229096 28059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:08.231606 28059 webserver.cc:460] Webserver started at http://127.8.218.194:32973/ using document root <none> and password file <none>
I20260928 10:49:08.232376 28059 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:08.232499 28059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:08.232923 28059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:08.236042 28059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "06f723a79b1d49bbb467e31ca55e07a5"
format_stamp: "Formatted at 2026-09-28 10:49:08 on dist-test-slave-mn38"
server_key: "627c65f3fc5db1d86e9baca3798ab5dc"
server_key_iv: "4f7b77efe0e3515c5ab30c768be58981"
server_key_version: "encryptionkey@0"
I20260928 10:49:08.236881 28059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "06f723a79b1d49bbb467e31ca55e07a5"
format_stamp: "Formatted at 2026-09-28 10:49:08 on dist-test-slave-mn38"
server_key: "627c65f3fc5db1d86e9baca3798ab5dc"
server_key_iv: "4f7b77efe0e3515c5ab30c768be58981"
server_key_version: "encryptionkey@0"
I20260928 10:49:08.242389 28059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 10:49:08.246327 28073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:08.248112 28059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:08.248298 28059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "06f723a79b1d49bbb467e31ca55e07a5"
format_stamp: "Formatted at 2026-09-28 10:49:08 on dist-test-slave-mn38"
server_key: "627c65f3fc5db1d86e9baca3798ab5dc"
server_key_iv: "4f7b77efe0e3515c5ab30c768be58981"
server_key_version: "encryptionkey@0"
I20260928 10:49:08.248519 28059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:08.286679 28059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:08.287539 28059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:08.287817 28059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:08.288858 28059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:08.290524 28059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:08.290617 28059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:08.290702 28059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:08.290755 28059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:08.321775 28059 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:39331
I20260928 10:49:08.321780 28185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:39331 every 8 connection(s)
I20260928 10:49:08.323215 28059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:49:08.324674 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 28059
I20260928 10:49:08.324862 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:49:08.325371 9067 external_mini_cluster.cc:1428] Setting key 48564fd9d6779bf244b1868953a09ff6
I20260928 10:49:08.328919 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37497
--builtin_ntp_servers=127.8.218.212:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:49:08.340808 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.326854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:41709 (local address 127.8.218.254:37497)
0928 10:49:08.327093 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:08.327101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:08.329465 (+ 2364us) server_negotiation.cc:407] Connection header received
0928 10:49:08.331790 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:08.331796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:08.331901 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:08.332015 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:08.333396 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.334620 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.336692 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.337244 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.337370 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.339708 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:08.339714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:08.339718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:08.339743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:08.339783 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:08.339797 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:08.339891 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:08.340293 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 10:49:08.340436 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:49:08.342505 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.327819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37497 (local address 127.8.218.194:41709)
0928 10:49:08.329210 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:08.329261 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:08.331364 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:08.332322 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:08.332332 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:08.332444 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:08.333186 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.333207 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.334905 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.334910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.336436 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.336458 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.337603 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.337612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.337988 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.339152 (+ 1164us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:08.339157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:08.339227 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:08.339384 (+ 157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:08.339930 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:08.339944 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:08.339955 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:49:08.340220 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 10:49:08.340781 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":367,"threads_started":1}
I20260928 10:49:08.344806 28186 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37497
I20260928 10:49:08.345213 28186 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:08.346232 28186 heartbeater.cc:511] Master 127.8.218.254:37497 requested a full tablet report, sending...
I20260928 10:49:08.348289 27869 ts_manager.cc:194] Registered new tserver with Master: 06f723a79b1d49bbb467e31ca55e07a5 (127.8.218.194:39331)
I20260928 10:49:08.349900 27869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.194:41709
W20260928 10:49:08.675277 28189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:08.675572 28189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:08.675657 28189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:08.683007 28189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:08.683113 28189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:08.683220 28189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:49:08.692523 28189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:37497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:08.693861 28189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:08.695778 28189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:08.710273 28194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:08.710289 28195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:08.710491 28197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:08.711163 28189 server_base.cc:1034] running on GCE node
I20260928 10:49:08.711802 28189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:08.712899 28189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:08.714124 28189 hybrid_clock.cc:648] HybridClock initialized: now 1790592548714055 us; error 62 us; skew 500 ppm
I20260928 10:49:08.714485 28189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:08.717062 28189 webserver.cc:460] Webserver started at http://127.8.218.195:34693/ using document root <none> and password file <none>
I20260928 10:49:08.717900 28189 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:08.718003 28189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:08.718320 28189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:08.721407 28189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "1626430dea8046ba93192c9d3ba16c9e"
format_stamp: "Formatted at 2026-09-28 10:49:08 on dist-test-slave-mn38"
server_key: "2df18d7c79ecb04d23034a0ec91e6b51"
server_key_iv: "42a158d21e29f35b557e365c3ed6acd5"
server_key_version: "encryptionkey@0"
I20260928 10:49:08.722329 28189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "1626430dea8046ba93192c9d3ba16c9e"
format_stamp: "Formatted at 2026-09-28 10:49:08 on dist-test-slave-mn38"
server_key: "2df18d7c79ecb04d23034a0ec91e6b51"
server_key_iv: "42a158d21e29f35b557e365c3ed6acd5"
server_key_version: "encryptionkey@0"
I20260928 10:49:08.727955 28189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 10:49:08.732138 28203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:08.734083 28189 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:49:08.734300 28189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "1626430dea8046ba93192c9d3ba16c9e"
format_stamp: "Formatted at 2026-09-28 10:49:08 on dist-test-slave-mn38"
server_key: "2df18d7c79ecb04d23034a0ec91e6b51"
server_key_iv: "42a158d21e29f35b557e365c3ed6acd5"
server_key_version: "encryptionkey@0"
I20260928 10:49:08.734635 28189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:08.750375 28189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:08.751314 28189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:08.751617 28189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:08.752705 28189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:08.754711 28189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:08.754817 28189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:08.754907 28189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:08.754987 28189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:08.793255 28189 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:35619
I20260928 10:49:08.793299 28315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:35619 every 8 connection(s)
I20260928 10:49:08.795012 28189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:49:08.802037 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 28189
I20260928 10:49:08.802243 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:49:08.802807 9067 external_mini_cluster.cc:1428] Setting key 07dba75653c69a6709296024e334417b
I20260928 10:49:08.812850 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.798153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:48651 (local address 127.8.218.254:37497)
0928 10:49:08.798361 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:08.798368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:08.800235 (+ 1867us) server_negotiation.cc:407] Connection header received
0928 10:49:08.803086 (+ 2851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:08.803095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:08.803196 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:08.803267 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:08.805057 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.806283 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.808782 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.809352 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.809527 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.811731 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:08.811740 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:08.811745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:08.811775 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:08.811825 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:08.811843 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:08.811942 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:08.812326 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 10:49:08.812441 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 10:49:08.814277 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.798827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:37497 (local address 127.8.218.195:48651)
0928 10:49:08.800030 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:08.800080 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:08.802677 (+ 2597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:08.803849 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:08.803859 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:08.804001 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:08.804825 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.804856 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.806553 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.806560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.808588 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.808613 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.809858 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.809867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.810253 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.811193 (+ 940us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:08.811197 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:08.811282 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:08.811415 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:08.811972 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:08.811987 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:08.811998 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:49:08.812261 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 10:49:08.812743 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":337,"threads_started":1}
I20260928 10:49:08.816231 28316 heartbeater.cc:347] Connected to a master server at 127.8.218.254:37497
I20260928 10:49:08.816582 28316 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:08.817479 28316 heartbeater.cc:511] Master 127.8.218.254:37497 requested a full tablet report, sending...
I20260928 10:49:08.819303 27869 ts_manager.cc:194] Registered new tserver with Master: 1626430dea8046ba93192c9d3ba16c9e (127.8.218.195:35619)
I20260928 10:49:08.820346 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:49:08.820523 27869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.195:48651
I20260928 10:49:08.831401 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.824640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.8.218.254:37497)
0928 10:49:08.824896 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:08.824903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:08.825193 (+ 290us) server_negotiation.cc:407] Connection header received
0928 10:49:08.825342 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:08.825346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:08.825421 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:08.825628 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:08.826165 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.827029 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.828650 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.829078 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.829197 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.830466 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:08.830472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:08.830475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:08.830498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:08.830538 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:08.830551 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:08.830684 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:08.830997 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 10:49:08.831082 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:49:08.836338 27869 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:49:08.844547 27869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:49:08.861717 28055 heartbeater.cc:503] Master 127.8.218.254:37497 was elected leader, sending a full tablet report...
I20260928 10:49:08.869530 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.858221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:39331 (local address 127.0.0.1:57832)
0928 10:49:08.860050 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:08.860090 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:08.860364 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:08.861494 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:08.861502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:08.861540 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:08.862080 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.862096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.864067 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.864072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.866500 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.866516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.868468 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.868474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.868684 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.868757 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 10:49:08.868903 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 10:49:08.869120 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":2895,"threads_started":1}
I20260928 10:49:08.869777 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.858388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.8.218.194:39331)
0928 10:49:08.859054 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:08.859064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:08.860210 (+ 1146us) server_negotiation.cc:407] Connection header received
0928 10:49:08.860608 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:08.860618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:08.860921 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:08.861120 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:08.862333 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.863865 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.866718 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.868273 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.868417 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.868698 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:08.869235 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 10:49:08.869430 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":313,"threads_started":1}
I20260928 10:49:08.873482 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.859307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.8.218.193:44929)
0928 10:49:08.860956 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:08.860965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:08.861597 (+ 632us) server_negotiation.cc:407] Connection header received
0928 10:49:08.861911 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:08.863352 (+ 1441us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:08.863654 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:08.863815 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:08.864804 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.867032 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.870276 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.871835 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.871972 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.872267 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:08.872880 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 10:49:08.873079 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1464,"thread_start_us":1338,"threads_started":1}
I20260928 10:49:08.874194 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.858768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44929 (local address 127.0.0.1:41510)
0928 10:49:08.861469 (+ 2701us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:08.861489 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:08.861708 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:08.864101 (+ 2393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:08.864105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:08.864133 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:08.864595 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.864610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.867315 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.867320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.869386 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.869397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.872413 (+ 3016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.872419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.872639 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.872676 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 10:49:08.873698 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0928 10:49:08.873851 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2497,"mutex_wait_us":4,"thread_start_us":1137,"threads_started":1}
I20260928 10:49:08.874146 28121 tablet_service.cc:1467] Processing CreateTablet for tablet 92f672a110c640ffa46b7c39848afcb0 (DEFAULT_TABLE table=test-table [id=3d6c800954db451bba48aa1977079c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:08.876168 28121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f672a110c640ffa46b7c39848afcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:08.877139 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.861548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:35619 (local address 127.0.0.1:41378)
0928 10:49:08.868083 (+ 6535us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:08.868104 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:08.868226 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:08.868895 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:08.868901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:08.868930 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:08.869294 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.869305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.871214 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.871219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.873928 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:08.873946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.876383 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.876392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:08.876582 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.876614 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:49:08.876693 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 10:49:08.876839 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6413,"thread_start_us":6264,"threads_started":1}
I20260928 10:49:08.878348 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:08.862554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.8.218.195:35619)
0928 10:49:08.867762 (+ 5208us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:08.867779 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:08.868216 (+ 437us) server_negotiation.cc:407] Connection header received
0928 10:49:08.868380 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:08.868389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:08.868675 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:08.868893 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:08.869549 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.871059 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.874227 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:08.875914 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:08.876045 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:08.877790 (+ 1745us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:08.877868 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 10:49:08.878040 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5034,"thread_start_us":4873,"threads_started":1}
I20260928 10:49:08.879696 27990 tablet_service.cc:1467] Processing CreateTablet for tablet 92f672a110c640ffa46b7c39848afcb0 (DEFAULT_TABLE table=test-table [id=3d6c800954db451bba48aa1977079c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:08.881280 27990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f672a110c640ffa46b7c39848afcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:08.882328 28251 tablet_service.cc:1467] Processing CreateTablet for tablet 92f672a110c640ffa46b7c39848afcb0 (DEFAULT_TABLE table=test-table [id=3d6c800954db451bba48aa1977079c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:08.883774 28251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f672a110c640ffa46b7c39848afcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:08.890937 28334 tablet_bootstrap.cc:492] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Bootstrap starting.
I20260928 10:49:08.893863 28335 tablet_bootstrap.cc:492] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: Bootstrap starting.
I20260928 10:49:08.896055 28334 tablet_bootstrap.cc:654] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:08.897135 28334 log.cc:824] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:08.897677 28335 tablet_bootstrap.cc:654] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:08.899046 28335 log.cc:824] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:08.899590 28337 tablet_bootstrap.cc:492] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Bootstrap starting.
I20260928 10:49:08.900676 28334 tablet_bootstrap.cc:492] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: No bootstrap required, opened a new log
I20260928 10:49:08.900996 28334 ts_tablet_manager.cc:1397] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 10:49:08.904220 28335 tablet_bootstrap.cc:492] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: No bootstrap required, opened a new log
I20260928 10:49:08.904539 28335 ts_tablet_manager.cc:1397] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260928 10:49:08.905408 28337 tablet_bootstrap.cc:654] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:08.906857 28337 log.cc:824] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:08.910602 28337 tablet_bootstrap.cc:492] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: No bootstrap required, opened a new log
I20260928 10:49:08.910931 28337 ts_tablet_manager.cc:1397] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260928 10:49:08.911463 28334 raft_consensus.cc:348] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:08.911962 28334 raft_consensus.cc:374] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:08.912081 28334 raft_consensus.cc:729] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f723a79b1d49bbb467e31ca55e07a5, State: Initialized, Role: FOLLOWER
I20260928 10:49:08.912364 28335 raft_consensus.cc:348] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:08.912775 28335 raft_consensus.cc:374] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:08.912880 28335 raft_consensus.cc:729] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1afdb0fbbaeb47fba02280922fe906a4, State: Initialized, Role: FOLLOWER
I20260928 10:49:08.912774 28334 consensus_queue.cc:260] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:08.913533 28335 consensus_queue.cc:260] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:08.914136 28186 heartbeater.cc:503] Master 127.8.218.254:37497 was elected leader, sending a full tablet report...
I20260928 10:49:08.914726 28334 ts_tablet_manager.cc:1428] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260928 10:49:08.917284 28335 ts_tablet_manager.cc:1428] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260928 10:49:08.923415 28337 raft_consensus.cc:348] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:08.923840 28337 raft_consensus.cc:374] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:08.923945 28337 raft_consensus.cc:729] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1626430dea8046ba93192c9d3ba16c9e, State: Initialized, Role: FOLLOWER
I20260928 10:49:08.924556 28337 consensus_queue.cc:260] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:08.925606 28316 heartbeater.cc:503] Master 127.8.218.254:37497 was elected leader, sending a full tablet report...
I20260928 10:49:08.926111 28337 ts_tablet_manager.cc:1428] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
W20260928 10:49:09.046932 28317 tablet.cc:2369] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:49:09.076056 28187 tablet.cc:2369] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:49:09.087548 28056 tablet.cc:2369] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:09.343986 28341 raft_consensus.cc:482] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:49:09.344304 28341 raft_consensus.cc:504] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:09.346287 28341 leader_election.cc:288] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f723a79b1d49bbb467e31ca55e07a5 (127.8.218.194:39331), 1626430dea8046ba93192c9d3ba16c9e (127.8.218.195:35619)
I20260928 10:49:09.353766 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.346790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:39331 (local address 127.8.218.193:36397)
0928 10:49:09.347376 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.347395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:09.347562 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:09.348007 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:09.348011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:09.348038 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:09.348417 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:09.348430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.349747 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.349753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:09.351545 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:09.351559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.352971 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.352977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:09.353169 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:09.353207 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 10:49:09.353308 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 10:49:09.353485 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":293,"threads_started":1}
I20260928 10:49:09.353808 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.346886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:36397 (local address 127.8.218.194:39331)
0928 10:49:09.347090 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:09.347097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:09.347508 (+ 411us) server_negotiation.cc:407] Connection header received
0928 10:49:09.347717 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:09.347723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:09.347831 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:09.347923 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:09.348596 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.349597 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.351783 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.352747 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.352893 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:09.352964 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:09.353365 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 10:49:09.353544 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:49:09.354138 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.347515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:35619 (local address 127.8.218.193:35609)
0928 10:49:09.348110 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.348127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:09.348231 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:09.348594 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:09.348599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:09.348627 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:09.348991 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:09.349003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.350413 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.350418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:09.351942 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:09.351957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.353386 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.353394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:09.353599 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:09.353633 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:49:09.353724 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 10:49:09.353863 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":249,"threads_started":1}
I20260928 10:49:09.354161 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.347759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:35609 (local address 127.8.218.195:35619)
0928 10:49:09.347946 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:09.347952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:09.348206 (+ 254us) server_negotiation.cc:407] Connection header received
0928 10:49:09.348344 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:09.348350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:09.348430 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:09.348532 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:09.349183 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.350212 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.352170 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.353121 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.353286 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:09.353353 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:09.353799 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 10:49:09.353940 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:49:09.355082 28271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f672a110c640ffa46b7c39848afcb0" candidate_uuid: "1afdb0fbbaeb47fba02280922fe906a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1626430dea8046ba93192c9d3ba16c9e" is_pre_election: true
I20260928 10:49:09.355074 28141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f672a110c640ffa46b7c39848afcb0" candidate_uuid: "1afdb0fbbaeb47fba02280922fe906a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f723a79b1d49bbb467e31ca55e07a5" is_pre_election: true
I20260928 10:49:09.355578 28271 raft_consensus.cc:2454] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1afdb0fbbaeb47fba02280922fe906a4 in term 0.
I20260928 10:49:09.355576 28141 raft_consensus.cc:2454] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1afdb0fbbaeb47fba02280922fe906a4 in term 0.
I20260928 10:49:09.356257 27944 leader_election.cc:302] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1626430dea8046ba93192c9d3ba16c9e, 1afdb0fbbaeb47fba02280922fe906a4; no voters:
I20260928 10:49:09.356704 28341 raft_consensus.cc:2781] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:49:09.356853 28341 raft_consensus.cc:482] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:49:09.356962 28341 raft_consensus.cc:3037] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:09.359391 28341 raft_consensus.cc:504] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:09.360013 28341 leader_election.cc:288] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [CANDIDATE]: Term 1 election: Requested vote from peers 06f723a79b1d49bbb467e31ca55e07a5 (127.8.218.194:39331), 1626430dea8046ba93192c9d3ba16c9e (127.8.218.195:35619)
I20260928 10:49:09.360610 28141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f672a110c640ffa46b7c39848afcb0" candidate_uuid: "1afdb0fbbaeb47fba02280922fe906a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f723a79b1d49bbb467e31ca55e07a5"
I20260928 10:49:09.360638 28271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f672a110c640ffa46b7c39848afcb0" candidate_uuid: "1afdb0fbbaeb47fba02280922fe906a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1626430dea8046ba93192c9d3ba16c9e"
I20260928 10:49:09.360855 28271 raft_consensus.cc:3037] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:09.360847 28141 raft_consensus.cc:3037] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:09.364028 28271 raft_consensus.cc:2454] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1afdb0fbbaeb47fba02280922fe906a4 in term 1.
I20260928 10:49:09.364029 28141 raft_consensus.cc:2454] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1afdb0fbbaeb47fba02280922fe906a4 in term 1.
I20260928 10:49:09.364512 27944 leader_election.cc:302] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f723a79b1d49bbb467e31ca55e07a5, 1afdb0fbbaeb47fba02280922fe906a4; no voters:
I20260928 10:49:09.364837 28341 raft_consensus.cc:2781] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:09.365257 28341 raft_consensus.cc:686] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 1 LEADER]: Becoming Leader. State: Replica: 1afdb0fbbaeb47fba02280922fe906a4, State: Running, Role: LEADER
I20260928 10:49:09.365828 28341 consensus_queue.cc:237] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } }
I20260928 10:49:09.370217 27868 catalog_manager.cc:5495] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1afdb0fbbaeb47fba02280922fe906a4 (127.8.218.193). New cstate: current_term: 1 leader_uuid: "1afdb0fbbaeb47fba02280922fe906a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1afdb0fbbaeb47fba02280922fe906a4" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 44929 } health_report { overall_health: HEALTHY } } }
I20260928 10:49:09.447211 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.440386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.8.218.193:44929)
0928 10:49:09.440891 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:09.440895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:09.440974 (+ 79us) server_negotiation.cc:407] Connection header received
0928 10:49:09.441185 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:09.441189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:09.441276 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:09.441455 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:09.442075 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.442908 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.444518 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.445033 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.445181 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:09.445980 (+ 799us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:09.446524 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:09.446693 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:09.446864 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 10:49:09.446913 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":264,"threads_started":1}
I20260928 10:49:09.456465 28141 raft_consensus.cc:1261] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 1 FOLLOWER]: Refusing update from remote peer 1afdb0fbbaeb47fba02280922fe906a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:09.456465 28271 raft_consensus.cc:1261] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 1 FOLLOWER]: Refusing update from remote peer 1afdb0fbbaeb47fba02280922fe906a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:09.457505 28341 consensus_queue.cc:1035] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f723a79b1d49bbb467e31ca55e07a5" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 39331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:09.457849 28345 consensus_queue.cc:1035] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1626430dea8046ba93192c9d3ba16c9e" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:09.466845 28353 mvcc.cc:205] Tried to move back new op lower bound from 7334267082562068480 to 7334267082206330880. Current Snapshot: MvccSnapshot[applied={T|T < 7334267082562068480}]
I20260928 10:49:09.483076 27869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:09.483325 27869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:09.487027 27869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e627a7f62d674ef3a4a7d90b47cc2484: Sending DeleteTablet for 3 replicas of tablet 92f672a110c640ffa46b7c39848afcb0
I20260928 10:49:09.488296 28251 tablet_service.cc:1514] Processing DeleteTablet for tablet 92f672a110c640ffa46b7c39848afcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:09 UTC) from {username='slave'} at 127.0.0.1:41378
I20260928 10:49:09.488274 28121 tablet_service.cc:1514] Processing DeleteTablet for tablet 92f672a110c640ffa46b7c39848afcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:09 UTC) from {username='slave'} at 127.0.0.1:57832
I20260928 10:49:09.489743 27990 tablet_service.cc:1514] Processing DeleteTablet for tablet 92f672a110c640ffa46b7c39848afcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:09 UTC) from {username='slave'} at 127.0.0.1:41510
I20260928 10:49:09.490037 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 27928
I20260928 10:49:09.500828 28365 tablet_replica.cc:330] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: stopping tablet replica
I20260928 10:49:09.501415 28365 raft_consensus.cc:2229] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:09.503446 28365 raft_consensus.cc:2258] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:09.506752 28365 ts_tablet_manager.cc:1905] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:09.511663 28366 tablet_replica.cc:330] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: stopping tablet replica
I20260928 10:49:09.512382 28366 raft_consensus.cc:2229] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:49:09.513311 28366 raft_consensus.cc:2258] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:09.516026 28366 ts_tablet_manager.cc:1905] T 92f672a110c640ffa46b7c39848afcb0 P 1afdb0fbbaeb47fba02280922fe906a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:09.516511 28364 tablet_replica.cc:330] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: stopping tablet replica
I20260928 10:49:09.517036 28364 raft_consensus.cc:2229] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:09.519860 28365 ts_tablet_manager.cc:1918] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:09.520079 28365 log.cc:1196] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/92f672a110c640ffa46b7c39848afcb0
I20260928 10:49:09.520172 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.491657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.8.218.193:44929)
0928 10:49:09.492501 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:09.492509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:09.506126 (+ 13617us) server_negotiation.cc:407] Connection header received
0928 10:49:09.506643 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:09.506649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:09.506807 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:09.506904 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:09.507626 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.508827 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.514001 (+ 5174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.514816 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.514956 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:09.518078 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:09.518093 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:09.518112 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:09.519070 (+ 958us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:09.519153 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:09.519189 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:09.519310 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:09.519628 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 10:49:09.519761 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695}
I20260928 10:49:09.520710 28365 ts_tablet_manager.cc:1939] T 92f672a110c640ffa46b7c39848afcb0 P 1626430dea8046ba93192c9d3ba16c9e: Deleting consensus metadata
I20260928 10:49:09.521839 28050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:09.522902 28364 raft_consensus.cc:2258] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:09.523301 27854 catalog_manager.cc:4841] TS 1626430dea8046ba93192c9d3ba16c9e (127.8.218.195:35619): tablet 92f672a110c640ffa46b7c39848afcb0 (table test-table [id=3d6c800954db451bba48aa1977079c63]) successfully deleted
I20260928 10:49:09.525334 28364 ts_tablet_manager.cc:1905] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:09.531047 28364 ts_tablet_manager.cc:1918] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:09.531242 28364 log.cc:1196] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/92f672a110c640ffa46b7c39848afcb0
I20260928 10:49:09.531659 28364 ts_tablet_manager.cc:1939] T 92f672a110c640ffa46b7c39848afcb0 P 06f723a79b1d49bbb467e31ca55e07a5: Deleting consensus metadata
I20260928 10:49:09.533416 27854 catalog_manager.cc:4841] TS 06f723a79b1d49bbb467e31ca55e07a5 (127.8.218.194:39331): tablet 92f672a110c640ffa46b7c39848afcb0 (table test-table [id=3d6c800954db451bba48aa1977079c63]) successfully deleted
I20260928 10:49:09.621524 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 27928
I20260928 10:49:09.637982 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 28059
W20260928 10:49:09.638028 27853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.218.193:44929 (error 108)
I20260928 10:49:09.640417 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.639501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44929 (local address 127.0.0.1:41550)
0928 10:49:09.640027 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.640219 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":242,"threads_started":1}
W20260928 10:49:09.640683 27853 catalog_manager.cc:4584] TS 1afdb0fbbaeb47fba02280922fe906a4 (127.8.218.193:44929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92f672a110c640ffa46b7c39848afcb0: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44929: connect: Connection refused (error 111)
I20260928 10:49:09.645634 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.638907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.8.218.194:39331)
0928 10:49:09.639105 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:09.639111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:09.639200 (+ 89us) server_negotiation.cc:407] Connection header received
0928 10:49:09.639395 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:09.639399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:09.639507 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:09.639601 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:09.640051 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.640912 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.642356 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.642809 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.642930 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:09.644048 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:09.644058 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:09.644074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:09.644741 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:09.644798 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:09.644822 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:09.644933 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:09.645088 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 10:49:09.645220 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:49:09.646379 28181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:09.694584 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.694106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44929 (local address 127.0.0.1:41558)
0928 10:49:09.694328 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.694430 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 10:49:09.695670 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.695261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44929 (local address 127.0.0.1:41568)
0928 10:49:09.695418 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.695497 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 10:49:09.738181 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 28059
I20260928 10:49:09.752671 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 28189
I20260928 10:49:09.756590 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.756143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44929 (local address 127.0.0.1:41570)
0928 10:49:09.756336 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.756432 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 10:49:09.757905 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.757405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44929 (local address 127.0.0.1:41578)
0928 10:49:09.757630 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.757732 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 10:49:09.760224 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.753633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.8.218.195:35619)
0928 10:49:09.753828 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:09.753833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:09.753852 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:49:09.753935 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:09.753939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:09.754045 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:09.754134 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:09.754626 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.755403 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.756853 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:09.757397 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:09.757570 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:09.758659 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:09.758670 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:09.758687 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:09.759307 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:09.759363 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:09.759427 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:09.759524 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:09.759683 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 10:49:09.759796 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 10:49:09.760952 28311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:09.838819 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.838191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44929 (local address 127.0.0.1:41580)
0928 10:49:09.838508 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.838625 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260928 10:49:09.840077 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:09.839571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:44929 (local address 127.0.0.1:41582)
0928 10:49:09.839802 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:09.839894 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.193:44929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 10:49:09.851719 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 28189
I20260928 10:49:09.865500 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 27838
I20260928 10:49:09.866648 27899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:09.978727 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 27838
2026-09-28T10:49:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3291 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:10 dist-test-slave-mn38 krb5kdc[28376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:10 dist-test-slave-mn38 krb5kdc[28376](info): set up 2 sockets
Sep 28 10:49:10 dist-test-slave-mn38 krb5kdc[28376](info): commencing operation
krb5kdc: starting...
W20260928 10:49:11.014657 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:11 dist-test-slave-mn38 krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:11.273238 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:44129
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:44129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:11.625730 28389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:11.626101 28389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:11.626211 28389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:11.633953 28389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:11.634081 28389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:11.634126 28389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:11.634181 28389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:11.634225 28389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:11.644646 28389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:44129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:11.646049 28389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:11.647542 28389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:11.660656 28394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:11.660640 28397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:11.660660 28395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:11.661238 28389 server_base.cc:1034] running on GCE node
I20260928 10:49:11.662119 28389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:11.663647 28389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:11.664866 28389 hybrid_clock.cc:648] HybridClock initialized: now 1790592551664806 us; error 51 us; skew 500 ppm
I20260928 10:49:11.665266 28389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:11 dist-test-slave-mn38 krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592551, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:11.668569 28389 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:11.670359 28389 webserver.cc:460] Webserver started at http://127.8.218.254:43131/ using document root <none> and password file <none>
I20260928 10:49:11.671113 28389 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:11.671244 28389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:11.671559 28389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:11.674762 28389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "695c1f0895b641f093b01362f99a2019"
format_stamp: "Formatted at 2026-09-28 10:49:11 on dist-test-slave-mn38"
server_key: "ec7b429c70ba554ae805caef34ab28cc"
server_key_iv: "2f888693d569e4c6d61da9670bf7d576"
server_key_version: "encryptionkey@0"
I20260928 10:49:11.675554 28389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "695c1f0895b641f093b01362f99a2019"
format_stamp: "Formatted at 2026-09-28 10:49:11 on dist-test-slave-mn38"
server_key: "ec7b429c70ba554ae805caef34ab28cc"
server_key_iv: "2f888693d569e4c6d61da9670bf7d576"
server_key_version: "encryptionkey@0"
I20260928 10:49:11.681124 28389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:49:11.684872 28404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:11.686383 28389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:49:11.686563 28389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "695c1f0895b641f093b01362f99a2019"
format_stamp: "Formatted at 2026-09-28 10:49:11 on dist-test-slave-mn38"
server_key: "ec7b429c70ba554ae805caef34ab28cc"
server_key_iv: "2f888693d569e4c6d61da9670bf7d576"
server_key_version: "encryptionkey@0"
I20260928 10:49:11.686770 28389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:11.707624 28389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:11.712658 28389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:11.713008 28389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:11.732318 28389 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:44129
I20260928 10:49:11.732324 28455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:44129 every 8 connection(s)
I20260928 10:49:11.733961 28389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:11.738288 28456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:11.738411 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 28389
I20260928 10:49:11.738582 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:11.739068 9067 external_mini_cluster.cc:1428] Setting key c65168b65a907f60c22fe0c51e8102e6
I20260928 10:49:11.753721 28456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019: Bootstrap starting.
Sep 28 10:49:11 dist-test-slave-mn38 krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:11.757766 28456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:11.759192 28456 log.cc:824] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:11.762928 28456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019: No bootstrap required, opened a new log
I20260928 10:49:11.764709 28459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:11.741046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.8.218.254:44129)
0928 10:49:11.741980 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:11.741998 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:11.742060 (+ 62us) server_negotiation.cc:407] Connection header received
0928 10:49:11.743786 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:11.743825 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:11.744435 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:11.744946 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:11.746451 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:11.748369 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:11.749910 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:11.751085 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:11.751286 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:11.755643 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:11.755657 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:11.755679 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:11.755739 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:11.759629 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:11.760621 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:11.760629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:11.760636 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:11.760850 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:11.761347 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:11.761352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:11.761355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:11.762125 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:11.762415 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:11.762813 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 10:49:11.763278 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":383,"threads_started":1}
I20260928 10:49:11.771315 28456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695c1f0895b641f093b01362f99a2019" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44129 } }
I20260928 10:49:11.771780 28456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:11.771876 28456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695c1f0895b641f093b01362f99a2019, State: Initialized, Role: FOLLOWER
I20260928 10:49:11.772516 28456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695c1f0895b641f093b01362f99a2019" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44129 } }
I20260928 10:49:11.772751 28456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:11.772881 28456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:11.773036 28456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:11.775271 28456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695c1f0895b641f093b01362f99a2019" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44129 } }
I20260928 10:49:11.775790 28456 leader_election.cc:302] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695c1f0895b641f093b01362f99a2019; no voters:
I20260928 10:49:11.776362 28456 leader_election.cc:288] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:11.776487 28461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:11.777122 28461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [term 1 LEADER]: Becoming Leader. State: Replica: 695c1f0895b641f093b01362f99a2019, State: Running, Role: LEADER
I20260928 10:49:11.777700 28461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695c1f0895b641f093b01362f99a2019" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44129 } }
I20260928 10:49:11.778411 28456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:11.780280 28462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695c1f0895b641f093b01362f99a2019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695c1f0895b641f093b01362f99a2019" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44129 } } }
I20260928 10:49:11.780651 28462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:11.781015 28463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695c1f0895b641f093b01362f99a2019. Latest consensus state: current_term: 1 leader_uuid: "695c1f0895b641f093b01362f99a2019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695c1f0895b641f093b01362f99a2019" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44129 } } }
I20260928 10:49:11.781330 28463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:11.783310 28466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:11.789080 28466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:11.798911 28466 catalog_manager.cc:1347] Generated new cluster ID: 332c692df49f4902b45975cab8616e82
I20260928 10:49:11.799052 28466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:11.815529 28466 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:11.817142 28466 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:11.826372 28466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019: Generated new TSK 0
I20260928 10:49:11.827711 28466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.218.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
Entry for principal kudu/127.8.218.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab.
I20260928 10:49:11.890959 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44129
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:12.238013 28482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:12.238348 28482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:12.238458 28482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:12.246261 28482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:12.246387 28482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:12.246537 28482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:49:12.257181 28482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.193.keytab
--principal=kudu/127.8.218.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:12.258580 28482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:12.259955 28482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:12.274523 28487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:12.274616 28490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:12.274556 28488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:12.275404 28482 server_base.cc:1034] running on GCE node
I20260928 10:49:12.276057 28482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:12.277169 28482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:12.278424 28482 hybrid_clock.cc:648] HybridClock initialized: now 1790592552278352 us; error 86 us; skew 500 ppm
I20260928 10:49:12.278784 28482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:12 dist-test-slave-mn38 krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592552, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:12.287292 28482 init.cc:377] Logged in from keytab as kudu/127.8.218.193@KRBTEST.COM (short username kudu)
I20260928 10:49:12.288975 28482 webserver.cc:460] Webserver started at http://127.8.218.193:42567/ using document root <none> and password file <none>
I20260928 10:49:12.289763 28482 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:12.289920 28482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:12.290306 28482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:12.293790 28482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "7312115bf65f496d8f3205890ec796a6"
format_stamp: "Formatted at 2026-09-28 10:49:12 on dist-test-slave-mn38"
server_key: "c3cc682cc9bba0e2928b9069593ad03f"
server_key_iv: "6acc58bb3cf5f71ae7097d0ef8e28069"
server_key_version: "encryptionkey@0"
I20260928 10:49:12.294660 28482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "7312115bf65f496d8f3205890ec796a6"
format_stamp: "Formatted at 2026-09-28 10:49:12 on dist-test-slave-mn38"
server_key: "c3cc682cc9bba0e2928b9069593ad03f"
server_key_iv: "6acc58bb3cf5f71ae7097d0ef8e28069"
server_key_version: "encryptionkey@0"
I20260928 10:49:12.302737 28482 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260928 10:49:12.307809 28497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:12.309801 28482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 10:49:12.310027 28482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "7312115bf65f496d8f3205890ec796a6"
format_stamp: "Formatted at 2026-09-28 10:49:12 on dist-test-slave-mn38"
server_key: "c3cc682cc9bba0e2928b9069593ad03f"
server_key_iv: "6acc58bb3cf5f71ae7097d0ef8e28069"
server_key_version: "encryptionkey@0"
I20260928 10:49:12.310359 28482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:12.333701 28482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:12.340157 28482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:12.340549 28482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:12.342048 28482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:12.344266 28482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:12.344408 28482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:12.344561 28482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:12.344630 28482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:12.377274 28482 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:41923
I20260928 10:49:12.377342 28609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:41923 every 8 connection(s)
I20260928 10:49:12.378609 28482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:49:12.387181 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 28482
I20260928 10:49:12.387356 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:49:12.387758 9067 external_mini_cluster.cc:1428] Setting key e9e64206e3918ac8b8a1ba437310fa15
Sep 28 10:49:12 dist-test-slave-mn38 krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592552, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.193@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:12.399698 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:12.381197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:33015 (local address 127.8.218.254:44129)
0928 10:49:12.381630 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:12.381634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:12.382583 (+ 949us) server_negotiation.cc:407] Connection header received
0928 10:49:12.384466 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:12.384472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:12.384556 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:12.384651 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:12.386245 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:12.386973 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:12.388819 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:12.389258 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:12.389374 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:12.393481 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:12.393490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:12.393496 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:12.393538 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:12.396313 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:12.397177 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:12.397182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:12.397184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:12.397320 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:12.397816 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:12.397820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:12.397822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:12.398104 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:12.398244 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:12.399203 (+ 959us) server_negotiation.cc:299] Negotiation successful
0928 10:49:12.399428 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260928 10:49:12.401062 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:12.381701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44129 (local address 127.8.218.193:33015)
0928 10:49:12.382418 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:12.382457 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:12.384178 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:12.384904 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:12.384913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:12.385491 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:12.386068 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:12.386087 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:12.387134 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:12.387137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:12.388662 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:12.388677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:12.389542 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:12.389551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:12.389788 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:12.390522 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:12.390552 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:12.393171 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:12.396509 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:12.396517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:12.396534 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:12.397025 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:12.397462 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:12.397467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:12.397471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:12.397687 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:12.398235 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:12.398244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:12.398722 (+ 478us) client_negotiation.cc:769] Sending connection context
0928 10:49:12.399033 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 10:49:12.399618 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":267,"threads_started":1}
I20260928 10:49:12.402853 28610 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44129
I20260928 10:49:12.403172 28610 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:12.403982 28610 heartbeater.cc:511] Master 127.8.218.254:44129 requested a full tablet report, sending...
I20260928 10:49:12.406273 28421 ts_manager.cc:194] Registered new tserver with Master: 7312115bf65f496d8f3205890ec796a6 (127.8.218.193:41923)
I20260928 10:49:12.407811 28421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.193@KRBTEST.COM'} at 127.8.218.193:33015
WARNING: no policy specified for kudu/127.8.218.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
Entry for principal kudu/127.8.218.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab.
I20260928 10:49:12.445799 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44129
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:12.784788 28616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:12.785106 28616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:12.785212 28616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:12.794647 28616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:12.794814 28616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:12.795034 28616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:49:12.805924 28616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.194.keytab
--principal=kudu/127.8.218.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:12.807329 28616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:12.808776 28616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:12.823529 28622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:12.823495 28624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:12.823563 28621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:12.824178 28616 server_base.cc:1034] running on GCE node
I20260928 10:49:12.824842 28616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:12.825930 28616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:12.827184 28616 hybrid_clock.cc:648] HybridClock initialized: now 1790592552827108 us; error 66 us; skew 500 ppm
I20260928 10:49:12.827565 28616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:12 dist-test-slave-mn38 krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592552, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:12.831168 28616 init.cc:377] Logged in from keytab as kudu/127.8.218.194@KRBTEST.COM (short username kudu)
I20260928 10:49:12.832811 28616 webserver.cc:460] Webserver started at http://127.8.218.194:42831/ using document root <none> and password file <none>
I20260928 10:49:12.833596 28616 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:12.833724 28616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:12.834043 28616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:12.837100 28616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "af18adbaed6d4499a3ae06648b6ca3be"
format_stamp: "Formatted at 2026-09-28 10:49:12 on dist-test-slave-mn38"
server_key: "232d72d7155e37072f5c49c6ae0a90f7"
server_key_iv: "4326ed1630cdec9cc2356e5348c65e29"
server_key_version: "encryptionkey@0"
I20260928 10:49:12.837960 28616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "af18adbaed6d4499a3ae06648b6ca3be"
format_stamp: "Formatted at 2026-09-28 10:49:12 on dist-test-slave-mn38"
server_key: "232d72d7155e37072f5c49c6ae0a90f7"
server_key_iv: "4326ed1630cdec9cc2356e5348c65e29"
server_key_version: "encryptionkey@0"
I20260928 10:49:12.843636 28616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 10:49:12.847772 28631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:12.849426 28616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:49:12.849675 28616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "af18adbaed6d4499a3ae06648b6ca3be"
format_stamp: "Formatted at 2026-09-28 10:49:12 on dist-test-slave-mn38"
server_key: "232d72d7155e37072f5c49c6ae0a90f7"
server_key_iv: "4326ed1630cdec9cc2356e5348c65e29"
server_key_version: "encryptionkey@0"
I20260928 10:49:12.849939 28616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:12.871367 28616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:12.876245 28616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:12.876559 28616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:12.877714 28616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:12.879287 28616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:12.879374 28616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:12.879508 28616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:12.879559 28616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:12.911893 28616 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:45993
I20260928 10:49:12.911926 28743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:45993 every 8 connection(s)
I20260928 10:49:12.913162 28616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:49:12.921607 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 28616
I20260928 10:49:12.921783 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:49:12.922256 9067 external_mini_cluster.cc:1428] Setting key 090758fd3f741d2d057663ec8420badd
Sep 28 10:49:12 dist-test-slave-mn38 krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592552, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.194@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:12.935619 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:12.915828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:59519 (local address 127.8.218.254:44129)
0928 10:49:12.916444 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:12.916468 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:12.917526 (+ 1058us) server_negotiation.cc:407] Connection header received
0928 10:49:12.919403 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:12.919409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:12.919495 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:12.919591 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:12.921321 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:12.922249 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:12.924604 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:12.925073 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:12.925178 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:12.929602 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:12.929608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:12.929612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:12.929652 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:12.932348 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:12.933171 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:12.933177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:12.933179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:12.933323 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:12.933815 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:12.933819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:12.933822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:12.934106 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:12.934247 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:12.934991 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 10:49:12.935231 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":384,"threads_started":1}
I20260928 10:49:12.936622 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:12.916492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44129 (local address 127.8.218.194:59519)
0928 10:49:12.917296 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:12.917340 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:12.919108 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:12.919845 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:12.919854 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:12.920401 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:12.921020 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:12.921048 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:12.922484 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:12.922490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:12.924453 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:12.924470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:12.925341 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:12.925348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:12.925682 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:12.926476 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:12.926515 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:12.929289 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:12.932531 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:12.932542 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:12.932562 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:12.933033 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:12.933463 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:12.933469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:12.933473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:12.933681 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:12.934239 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:12.934248 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:12.934614 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 10:49:12.934865 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 10:49:12.935444 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":358,"threads_started":1}
I20260928 10:49:12.938117 28744 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44129
I20260928 10:49:12.938410 28744 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:12.939121 28744 heartbeater.cc:511] Master 127.8.218.254:44129 requested a full tablet report, sending...
I20260928 10:49:12.940569 28421 ts_manager.cc:194] Registered new tserver with Master: af18adbaed6d4499a3ae06648b6ca3be (127.8.218.194:45993)
I20260928 10:49:12.941390 28421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.194@KRBTEST.COM'} at 127.8.218.194:59519
WARNING: no policy specified for kudu/127.8.218.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
Entry for principal kudu/127.8.218.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab.
I20260928 10:49:12.977856 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44129
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:13.320288 28750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:13.320609 28750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:13.320706 28750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:13.328374 28750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:13.328517 28750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:13.328636 28750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:49:13.339218 28750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.195.keytab
--principal=kudu/127.8.218.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:44129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:13.340516 28750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:13.341887 28750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:13.356145 28756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:13.356181 28758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:13.356216 28755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:13.356993 28750 server_base.cc:1034] running on GCE node
I20260928 10:49:13.357725 28750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:13.358842 28750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:13.360067 28750 hybrid_clock.cc:648] HybridClock initialized: now 1790592553360011 us; error 49 us; skew 500 ppm
I20260928 10:49:13.360390 28750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:13 dist-test-slave-mn38 krb5kdc[28376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592553, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:13.363677 28750 init.cc:377] Logged in from keytab as kudu/127.8.218.195@KRBTEST.COM (short username kudu)
I20260928 10:49:13.365208 28750 webserver.cc:460] Webserver started at http://127.8.218.195:36145/ using document root <none> and password file <none>
I20260928 10:49:13.365954 28750 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:13.366074 28750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:13.366375 28750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:13.369324 28750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "5acbe1bd4ceb4e08b75414e72c7e4584"
format_stamp: "Formatted at 2026-09-28 10:49:13 on dist-test-slave-mn38"
server_key: "a0fe1edfd4af7857d23579227065580a"
server_key_iv: "e99cb86328a57a6a9a6222b1c1a6c844"
server_key_version: "encryptionkey@0"
I20260928 10:49:13.370170 28750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "5acbe1bd4ceb4e08b75414e72c7e4584"
format_stamp: "Formatted at 2026-09-28 10:49:13 on dist-test-slave-mn38"
server_key: "a0fe1edfd4af7857d23579227065580a"
server_key_iv: "e99cb86328a57a6a9a6222b1c1a6c844"
server_key_version: "encryptionkey@0"
I20260928 10:49:13.375599 28750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:49:13.379542 28765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:13.381071 28750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:49:13.381229 28750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "5acbe1bd4ceb4e08b75414e72c7e4584"
format_stamp: "Formatted at 2026-09-28 10:49:13 on dist-test-slave-mn38"
server_key: "a0fe1edfd4af7857d23579227065580a"
server_key_iv: "e99cb86328a57a6a9a6222b1c1a6c844"
server_key_version: "encryptionkey@0"
I20260928 10:49:13.381417 28750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:13.397063 28750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:13.401934 28750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:13.402226 28750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:13.403292 28750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:13.404955 28750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:13.405028 28750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:13.405118 28750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:13.405164 28750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:13.411355 28610 heartbeater.cc:503] Master 127.8.218.254:44129 was elected leader, sending a full tablet report...
I20260928 10:49:13.437366 28750 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:42801
I20260928 10:49:13.437475 28877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:42801 every 8 connection(s)
I20260928 10:49:13.438774 28750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:49:13.443512 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 28750
I20260928 10:49:13.443677 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:49:13.444105 9067 external_mini_cluster.cc:1428] Setting key 8ad434f5fe85527df81f53085a4f7220
Sep 28 10:49:13 dist-test-slave-mn38 krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592553, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.195@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:13.460597 28880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.441562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:38513 (local address 127.8.218.254:44129)
0928 10:49:13.442038 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.442043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.443082 (+ 1039us) server_negotiation.cc:407] Connection header received
0928 10:49:13.445147 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.445153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.445258 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.445358 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:13.447395 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.448189 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.449844 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.450258 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.450367 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.453971 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:13.453977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:13.453980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:13.454027 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:13.456833 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:13.457880 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:13.457891 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:13.457895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:13.458067 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:13.458631 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:13.458636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:13.458638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:13.459066 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:13.459213 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.460109 (+ 896us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.460309 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":222,"threads_started":1}
I20260928 10:49:13.461529 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.442150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44129 (local address 127.8.218.195:38513)
0928 10:49:13.442904 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:13.442945 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:13.444856 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:13.445715 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:13.445728 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:13.446448 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:13.447211 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.447236 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.448323 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.448327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.449699 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.449713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.450484 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.450489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.450709 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.451351 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:13.451374 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:13.453688 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:13.457014 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:13.457023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:13.457038 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:13.457608 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:13.458219 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:13.458225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:13.458229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:13.458504 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:13.459182 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:13.459195 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:13.459644 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 10:49:13.459931 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 10:49:13.460362 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":280,"threads_started":1}
I20260928 10:49:13.463023 28878 heartbeater.cc:347] Connected to a master server at 127.8.218.254:44129
I20260928 10:49:13.463347 28878 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:13.464097 28878 heartbeater.cc:511] Master 127.8.218.254:44129 requested a full tablet report, sending...
I20260928 10:49:13.465674 28421 ts_manager.cc:194] Registered new tserver with Master: 5acbe1bd4ceb4e08b75414e72c7e4584 (127.8.218.195:42801)
I20260928 10:49:13.466544 28421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.218.195@KRBTEST.COM'} at 127.8.218.195:38513
I20260928 10:49:13.472131 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:49:13.488487 28880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.476241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.8.218.254:44129)
0928 10:49:13.476417 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.476421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.476809 (+ 388us) server_negotiation.cc:407] Connection header received
0928 10:49:13.477024 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.477028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.477108 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.477200 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:13.478599 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.479371 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.480637 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.481056 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.481149 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.483444 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:13.483449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:13.483453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:13.483495 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:13.486113 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:13.486691 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:13.486694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:13.486697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:13.486805 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:13.487180 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:13.487184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:13.487186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:13.487529 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:13.487731 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.488053 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.488250 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:49:13.492774 28421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:49:13.499867 28421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:49:13.525347 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.512590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:45993 (local address 127.0.0.1:33422)
0928 10:49:13.513426 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:13.513501 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:13.513699 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:13.515616 (+ 1917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:13.515624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:13.515659 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:13.516125 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.516139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.518698 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.518705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.521684 (+ 2979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.521709 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.524105 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.524112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.524396 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.524473 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 10:49:13.524622 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 10:49:13.524874 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":33,"thread_start_us":418,"threads_started":1}
I20260928 10:49:13.525784 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.512733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.8.218.194:45993)
0928 10:49:13.514857 (+ 2124us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.514866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.514901 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:49:13.515013 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.515022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.515354 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.516373 (+ 1019us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:13.516654 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.518517 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.521994 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.523830 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.524003 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.524264 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.525186 (+ 922us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.525399 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1941,"thread_start_us":1825,"threads_started":1}
I20260928 10:49:13.527968 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.513329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.8.218.193:41923)
0928 10:49:13.514362 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.514372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.515330 (+ 958us) server_negotiation.cc:407] Connection header received
0928 10:49:13.515668 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.515677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.515918 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.516097 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:13.517312 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.519007 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.524568 (+ 5561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.526265 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.526467 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.526816 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.527446 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.527639 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":669,"threads_started":1}
I20260928 10:49:13.528774 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.512697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41923 (local address 127.0.0.1:54888)
0928 10:49:13.515210 (+ 2513us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:13.515227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:13.515461 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:13.516454 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:13.516460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:13.516490 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:13.517034 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.517049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.519225 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.519231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.524017 (+ 4786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.524038 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.527034 (+ 2996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.527040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.527219 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.527253 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 10:49:13.528249 (+ 996us) client_negotiation.cc:240] Negotiation successful
0928 10:49:13.528420 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2379,"mutex_wait_us":2318,"thread_start_us":2200,"threads_started":1}
I20260928 10:49:13.530686 28679 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8cf2bffcbb401f82bbacc770b6ec65 (DEFAULT_TABLE table=test-table [id=08b56be1379f40e9be36b27f66a49f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:13.532383 28679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8cf2bffcbb401f82bbacc770b6ec65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:13.533200 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.515412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:42801 (local address 127.0.0.1:40830)
0928 10:49:13.524617 (+ 9205us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:13.524641 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:13.524795 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:13.525909 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:13.525914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:13.525947 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:13.526570 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.526589 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.528696 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.528706 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.530562 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.530579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.532379 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.532390 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.532558 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.532593 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 10:49:13.532704 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 10:49:13.532856 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9049,"thread_start_us":9467,"threads_started":1}
I20260928 10:49:13.533164 28545 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8cf2bffcbb401f82bbacc770b6ec65 (DEFAULT_TABLE table=test-table [id=08b56be1379f40e9be36b27f66a49f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:13.534054 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.515747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.8.218.195:42801)
0928 10:49:13.524798 (+ 9051us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.524809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.524849 (+ 40us) server_negotiation.cc:407] Connection header received
0928 10:49:13.524958 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.524966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.525302 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.525591 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:13.526913 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.528531 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.530763 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.531844 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.531981 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.532225 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.533602 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.533761 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8848,"thread_start_us":11067,"threads_started":1}
I20260928 10:49:13.534905 28545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8cf2bffcbb401f82bbacc770b6ec65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:13.537211 28813 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8cf2bffcbb401f82bbacc770b6ec65 (DEFAULT_TABLE table=test-table [id=08b56be1379f40e9be36b27f66a49f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:13.539072 28813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8cf2bffcbb401f82bbacc770b6ec65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:13.548090 28897 tablet_bootstrap.cc:492] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Bootstrap starting.
I20260928 10:49:13.551250 28898 tablet_bootstrap.cc:492] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Bootstrap starting.
I20260928 10:49:13.552371 28897 tablet_bootstrap.cc:654] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:13.552989 28899 tablet_bootstrap.cc:492] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Bootstrap starting.
I20260928 10:49:13.554031 28897 log.cc:824] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:13.555337 28898 tablet_bootstrap.cc:654] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:13.556545 28899 tablet_bootstrap.cc:654] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:13.556638 28898 log.cc:824] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:13.557837 28897 tablet_bootstrap.cc:492] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: No bootstrap required, opened a new log
I20260928 10:49:13.557811 28899 log.cc:824] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:13.558169 28897 ts_tablet_manager.cc:1397] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 10:49:13.560528 28898 tablet_bootstrap.cc:492] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: No bootstrap required, opened a new log
I20260928 10:49:13.560848 28898 ts_tablet_manager.cc:1397] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 10:49:13.563515 28899 tablet_bootstrap.cc:492] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: No bootstrap required, opened a new log
I20260928 10:49:13.563807 28899 ts_tablet_manager.cc:1397] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 10:49:13.566046 28897 raft_consensus.cc:348] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.566522 28897 raft_consensus.cc:374] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:13.566620 28897 raft_consensus.cc:729] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af18adbaed6d4499a3ae06648b6ca3be, State: Initialized, Role: FOLLOWER
I20260928 10:49:13.567291 28897 consensus_queue.cc:260] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.568452 28744 heartbeater.cc:503] Master 127.8.218.254:44129 was elected leader, sending a full tablet report...
I20260928 10:49:13.569232 28897 ts_tablet_manager.cc:1428] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 10:49:13.571447 28898 raft_consensus.cc:348] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.571947 28898 raft_consensus.cc:374] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:13.572052 28898 raft_consensus.cc:729] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7312115bf65f496d8f3205890ec796a6, State: Initialized, Role: FOLLOWER
I20260928 10:49:13.572746 28898 consensus_queue.cc:260] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.574877 28898 ts_tablet_manager.cc:1428] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Time spent starting tablet: real 0.014s user 0.004s sys 0.008s
I20260928 10:49:13.574709 28899 raft_consensus.cc:348] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.575212 28899 raft_consensus.cc:374] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:13.575312 28899 raft_consensus.cc:729] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5acbe1bd4ceb4e08b75414e72c7e4584, State: Initialized, Role: FOLLOWER
I20260928 10:49:13.576038 28899 consensus_queue.cc:260] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.577116 28878 heartbeater.cc:503] Master 127.8.218.254:44129 was elected leader, sending a full tablet report...
I20260928 10:49:13.577913 28899 ts_tablet_manager.cc:1428] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
W20260928 10:49:13.631146 28611 tablet.cc:2369] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:49:13.665548 28745 tablet.cc:2369] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 10:49:13.690557 28879 tablet.cc:2369] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:13.713889 28905 raft_consensus.cc:482] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:49:13.714197 28905 raft_consensus.cc:504] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.716140 28905 leader_election.cc:288] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af18adbaed6d4499a3ae06648b6ca3be (127.8.218.194:45993), 7312115bf65f496d8f3205890ec796a6 (127.8.218.193:41923)
I20260928 10:49:13.721992 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.716558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:58117 (local address 127.8.218.194:45993)
0928 10:49:13.716752 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.716758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.716929 (+ 171us) server_negotiation.cc:407] Connection header received
0928 10:49:13.717168 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.717174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.717264 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.717377 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:13.717849 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.718569 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.720319 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.721078 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.721226 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.721315 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.721607 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.721739 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 10:49:13.721997 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.716411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:45993 (local address 127.8.218.195:58117)
0928 10:49:13.716805 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:13.716830 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:13.716970 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:13.717415 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:13.717466 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:13.717498 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:13.717702 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.717713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.718701 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.718707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.720142 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.720155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.721231 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.721236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.721412 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.721466 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 10:49:13.721552 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 10:49:13.721694 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260928 10:49:13.722788 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.716671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:50753 (local address 127.8.218.193:41923)
0928 10:49:13.716888 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.716893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.717534 (+ 641us) server_negotiation.cc:407] Connection header received
0928 10:49:13.717773 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.717778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.717852 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.717936 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:13.718614 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.719356 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.721029 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.721875 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.722023 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.722094 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.722422 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.722548 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:49:13.722844 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.716642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:41923 (local address 127.8.218.195:50753)
0928 10:49:13.717415 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:13.717456 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:13.717625 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:13.717974 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:13.717978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:13.718005 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:13.718305 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.718314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.719513 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.719517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.720868 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:13.720881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.722076 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.722083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:13.722266 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.722297 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:49:13.722378 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 10:49:13.722508 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":451,"threads_started":1}
I20260928 10:49:13.723209 28699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8cf2bffcbb401f82bbacc770b6ec65" candidate_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af18adbaed6d4499a3ae06648b6ca3be" is_pre_election: true
I20260928 10:49:13.723824 28699 raft_consensus.cc:2454] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5acbe1bd4ceb4e08b75414e72c7e4584 in term 0.
I20260928 10:49:13.723811 28565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8cf2bffcbb401f82bbacc770b6ec65" candidate_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7312115bf65f496d8f3205890ec796a6" is_pre_election: true
I20260928 10:49:13.724350 28565 raft_consensus.cc:2454] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5acbe1bd4ceb4e08b75414e72c7e4584 in term 0.
I20260928 10:49:13.724573 28768 leader_election.cc:302] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5acbe1bd4ceb4e08b75414e72c7e4584, af18adbaed6d4499a3ae06648b6ca3be; no voters:
I20260928 10:49:13.725010 28905 raft_consensus.cc:2781] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:49:13.725153 28905 raft_consensus.cc:482] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:49:13.725252 28905 raft_consensus.cc:3037] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:13.727468 28905 raft_consensus.cc:504] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.728092 28905 leader_election.cc:288] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [CANDIDATE]: Term 1 election: Requested vote from peers af18adbaed6d4499a3ae06648b6ca3be (127.8.218.194:45993), 7312115bf65f496d8f3205890ec796a6 (127.8.218.193:41923)
I20260928 10:49:13.728674 28699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8cf2bffcbb401f82bbacc770b6ec65" candidate_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af18adbaed6d4499a3ae06648b6ca3be"
I20260928 10:49:13.728674 28565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8cf2bffcbb401f82bbacc770b6ec65" candidate_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7312115bf65f496d8f3205890ec796a6"
I20260928 10:49:13.728919 28699 raft_consensus.cc:3037] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:13.728966 28565 raft_consensus.cc:3037] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:13.732427 28565 raft_consensus.cc:2454] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5acbe1bd4ceb4e08b75414e72c7e4584 in term 1.
I20260928 10:49:13.732427 28699 raft_consensus.cc:2454] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5acbe1bd4ceb4e08b75414e72c7e4584 in term 1.
I20260928 10:49:13.733054 28768 leader_election.cc:302] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5acbe1bd4ceb4e08b75414e72c7e4584, 7312115bf65f496d8f3205890ec796a6, af18adbaed6d4499a3ae06648b6ca3be; no voters:
I20260928 10:49:13.733495 28905 raft_consensus.cc:2781] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:13.733999 28905 raft_consensus.cc:686] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 1 LEADER]: Becoming Leader. State: Replica: 5acbe1bd4ceb4e08b75414e72c7e4584, State: Running, Role: LEADER
I20260928 10:49:13.734575 28905 consensus_queue.cc:237] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } }
I20260928 10:49:13.740099 28421 catalog_manager.cc:5495] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5acbe1bd4ceb4e08b75414e72c7e4584 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5acbe1bd4ceb4e08b75414e72c7e4584" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 42801 } health_report { overall_health: HEALTHY } } }
I20260928 10:49:13.768136 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.762173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.8.218.195:42801)
0928 10:49:13.762414 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.762421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.762508 (+ 87us) server_negotiation.cc:407] Connection header received
0928 10:49:13.762824 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.762829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.762918 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.763057 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:13.763443 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.764210 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.765751 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.766180 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.766298 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.766921 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:13.767483 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:13.767635 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.767765 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.767865 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:49:13.777479 28699 raft_consensus.cc:1261] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 1 FOLLOWER]: Refusing update from remote peer 5acbe1bd4ceb4e08b75414e72c7e4584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:13.777493 28565 raft_consensus.cc:1261] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 1 FOLLOWER]: Refusing update from remote peer 5acbe1bd4ceb4e08b75414e72c7e4584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:13.778460 28907 consensus_queue.cc:1035] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af18adbaed6d4499a3ae06648b6ca3be" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 45993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:13.778826 28905 consensus_queue.cc:1035] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7312115bf65f496d8f3205890ec796a6" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 41923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:13.792622 28912 mvcc.cc:205] Tried to move back new op lower bound from 7334267100261806080 to 7334267100101898240. Current Snapshot: MvccSnapshot[applied={T|T < 7334267100261806080}]
I20260928 10:49:13.810976 28420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:13.811340 28420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:13.815529 28420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 695c1f0895b641f093b01362f99a2019: Sending DeleteTablet for 3 replicas of tablet 1d8cf2bffcbb401f82bbacc770b6ec65
I20260928 10:49:13.816897 28679 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8cf2bffcbb401f82bbacc770b6ec65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:13 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:33422
I20260928 10:49:13.817415 28545 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8cf2bffcbb401f82bbacc770b6ec65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:13 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:54888
I20260928 10:49:13.818401 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 28482
I20260928 10:49:13.818423 28813 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8cf2bffcbb401f82bbacc770b6ec65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:13 UTC) from {username='kudu', principal='kudu/127.8.218.254@KRBTEST.COM'} at 127.0.0.1:40830
I20260928 10:49:13.828719 28920 tablet_replica.cc:330] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: stopping tablet replica
I20260928 10:49:13.828961 28919 tablet_replica.cc:330] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: stopping tablet replica
I20260928 10:49:13.829587 28919 raft_consensus.cc:2229] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:13.829618 28920 raft_consensus.cc:2229] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:13.830253 28919 raft_consensus.cc:2258] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:13.830320 28920 raft_consensus.cc:2258] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:13.832912 28920 ts_tablet_manager.cc:1905] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:13.832912 28919 ts_tablet_manager.cc:1905] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:13.834532 28921 tablet_replica.cc:330] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: stopping tablet replica
I20260928 10:49:13.835054 28921 raft_consensus.cc:2229] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:49:13.835860 28921 raft_consensus.cc:2258] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 10:49:13 dist-test-slave-mn38 krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.193@KRBTEST.COM
I20260928 10:49:13.838569 28921 ts_tablet_manager.cc:1905] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:13.841367 28920 ts_tablet_manager.cc:1918] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:13.841746 28920 log.cc:1196] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/1d8cf2bffcbb401f82bbacc770b6ec65
I20260928 10:49:13.842386 28920 ts_tablet_manager.cc:1939] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 7312115bf65f496d8f3205890ec796a6: Deleting consensus metadata
I20260928 10:49:13.845110 28406 catalog_manager.cc:4841] TS 7312115bf65f496d8f3205890ec796a6 (127.8.218.193:41923): tablet 1d8cf2bffcbb401f82bbacc770b6ec65 (table test-table [id=08b56be1379f40e9be36b27f66a49f59]) successfully deleted
I20260928 10:49:13.847054 28921 ts_tablet_manager.cc:1918] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:13.847252 28921 log.cc:1196] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/1d8cf2bffcbb401f82bbacc770b6ec65
I20260928 10:49:13.847457 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.819867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.8.218.193:41923)
0928 10:49:13.820076 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.820083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.826066 (+ 5983us) server_negotiation.cc:407] Connection header received
0928 10:49:13.826271 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.826279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.826392 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.826499 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:13.828482 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.829823 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.831739 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.832404 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.832526 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.838413 (+ 5887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:13.838432 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:13.838450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:13.838500 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:13.842777 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:13.844277 (+ 1500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:13.844289 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:13.844292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:13.844514 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:13.845125 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:13.845130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:13.845133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:13.845737 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:13.846063 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.846721 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.846968 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:49:13.847821 28921 ts_tablet_manager.cc:1939] T 1d8cf2bffcbb401f82bbacc770b6ec65 P 5acbe1bd4ceb4e08b75414e72c7e4584: Deleting consensus metadata
I20260928 10:49:13.848146 28605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:13.853154 28919 ts_tablet_manager.cc:1918] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:13.853318 28919 log.cc:1196] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/1d8cf2bffcbb401f82bbacc770b6ec65
I20260928 10:49:13.853768 28919 ts_tablet_manager.cc:1939] T 1d8cf2bffcbb401f82bbacc770b6ec65 P af18adbaed6d4499a3ae06648b6ca3be: Deleting consensus metadata
I20260928 10:49:13.855553 28407 catalog_manager.cc:4841] TS af18adbaed6d4499a3ae06648b6ca3be (127.8.218.194:45993): tablet 1d8cf2bffcbb401f82bbacc770b6ec65 (table test-table [id=08b56be1379f40e9be36b27f66a49f59]) successfully deleted
I20260928 10:49:13.856077 28406 catalog_manager.cc:4841] TS 5acbe1bd4ceb4e08b75414e72c7e4584 (127.8.218.195:42801): tablet 1d8cf2bffcbb401f82bbacc770b6ec65 (table test-table [id=08b56be1379f40e9be36b27f66a49f59]) successfully deleted
I20260928 10:49:13.940858 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 28482
I20260928 10:49:13.956707 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 28616
Sep 28 10:49:13 dist-test-slave-mn38 krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.194@KRBTEST.COM
I20260928 10:49:13.974269 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:13.957606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.8.218.194:45993)
0928 10:49:13.957815 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:13.957821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:13.957843 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:49:13.957910 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:13.957916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:13.958006 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:13.958129 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:13.959439 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.960580 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.961994 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:13.962639 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:13.962777 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:13.967055 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:13.967078 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:13.967099 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:13.967162 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:13.971030 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:13.971928 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:13.971938 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:13.971942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:13.972106 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:13.972479 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:13.972484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:13.972488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:13.973001 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:13.973322 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:13.973624 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 10:49:13.973868 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:49:13.975096 28739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:14.065524 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 28616
I20260928 10:49:14.080165 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 28750
Sep 28 10:49:14 dist-test-slave-mn38 krb5kdc[28376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.195@KRBTEST.COM
I20260928 10:49:14.097862 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:14.080883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.8.218.195:42801)
0928 10:49:14.081051 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:14.081058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:14.081129 (+ 71us) server_negotiation.cc:407] Connection header received
0928 10:49:14.081317 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:14.081321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:14.081406 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:14.081541 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:14.082786 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:14.083583 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:14.084922 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:14.085421 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:14.085577 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:14.090717 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:14.090731 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:14.090747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:14.090805 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:14.094347 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:14.095378 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:14.095386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:14.095389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:14.095593 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:14.096060 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:14.096063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:14.096065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:14.096454 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:14.096703 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:14.097117 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 10:49:14.097349 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:49:14.098824 28873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:14.221391 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 28750
I20260928 10:49:14.237572 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 28389
I20260928 10:49:14.238690 28451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:14.352151 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 28389
2026-09-28T10:49:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4374 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T10:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:14Z Disabled control of system clock
I20260928 10:49:14.426339 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:40013
--webserver_interface=127.8.218.254
--webserver_port=0
--builtin_ntp_servers=127.8.218.212:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:40013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:49:14.780998 28930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:14.781288 28930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:14.781371 28930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:14.789969 28930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:14.790127 28930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:14.790194 28930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:14.790277 28930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:14.790340 28930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:14.801585 28930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:40013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:14.802919 28930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:14.804770 28930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:14.817850 28936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:14.817899 28938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:14.817878 28935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:14.818955 28930 server_base.cc:1034] running on GCE node
I20260928 10:49:14.819936 28930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:14.821679 28930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:14.822940 28930 hybrid_clock.cc:648] HybridClock initialized: now 1790592554822885 us; error 41 us; skew 500 ppm
I20260928 10:49:14.823343 28930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:14.826061 28930 webserver.cc:460] Webserver started at http://127.8.218.254:37509/ using document root <none> and password file <none>
I20260928 10:49:14.826870 28930 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:14.827006 28930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:14.827338 28930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:14.830649 28930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "b665c098fb29424a80ad18b28a5b9daf"
format_stamp: "Formatted at 2026-09-28 10:49:14 on dist-test-slave-mn38"
server_key: "e3c35fd02fc63a30eb81e710a2cb25a8"
server_key_iv: "00b1685b3c3454846675f35d7819ff37"
server_key_version: "encryptionkey@0"
I20260928 10:49:14.831579 28930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "b665c098fb29424a80ad18b28a5b9daf"
format_stamp: "Formatted at 2026-09-28 10:49:14 on dist-test-slave-mn38"
server_key: "e3c35fd02fc63a30eb81e710a2cb25a8"
server_key_iv: "00b1685b3c3454846675f35d7819ff37"
server_key_version: "encryptionkey@0"
I20260928 10:49:14.837512 28930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:49:14.842272 28944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:14.843926 28930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:49:14.844141 28930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "b665c098fb29424a80ad18b28a5b9daf"
format_stamp: "Formatted at 2026-09-28 10:49:14 on dist-test-slave-mn38"
server_key: "e3c35fd02fc63a30eb81e710a2cb25a8"
server_key_iv: "00b1685b3c3454846675f35d7819ff37"
server_key_version: "encryptionkey@0"
I20260928 10:49:14.844383 28930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:14.878280 28930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:14.879144 28930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:14.879423 28930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:14.897651 28930 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:40013
I20260928 10:49:14.897681 28995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:40013 every 8 connection(s)
I20260928 10:49:14.899178 28930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:14.902256 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 28930
I20260928 10:49:14.902423 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:14.902832 9067 external_mini_cluster.cc:1428] Setting key c9e975fa05ec101ac1abcd3a88e10f82
I20260928 10:49:14.903590 28996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:14.919554 28996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf: Bootstrap starting.
I20260928 10:49:14.920018 28999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:14.904853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.8.218.254:40013)
0928 10:49:14.906045 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:14.906060 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:14.906115 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:49:14.908550 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:14.908586 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:14.909320 (+ 734us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:14.909876 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:14.910539 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:14.912551 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:14.914237 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:14.915342 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:14.915578 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:14.916898 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:14.916910 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:14.916933 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:14.917751 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:14.917859 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:14.917894 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:14.918019 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:14.918268 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 10:49:14.918651 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":364,"threads_started":1}
I20260928 10:49:14.923229 28996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:14.924408 28996 log.cc:824] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:14.928269 28996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf: No bootstrap required, opened a new log
I20260928 10:49:14.936988 28996 raft_consensus.cc:348] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b665c098fb29424a80ad18b28a5b9daf" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 40013 } }
I20260928 10:49:14.937538 28996 raft_consensus.cc:374] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:14.937664 28996 raft_consensus.cc:729] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b665c098fb29424a80ad18b28a5b9daf, State: Initialized, Role: FOLLOWER
I20260928 10:49:14.938323 28996 consensus_queue.cc:260] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b665c098fb29424a80ad18b28a5b9daf" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 40013 } }
I20260928 10:49:14.938549 28996 raft_consensus.cc:388] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:14.938688 28996 raft_consensus.cc:482] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:14.938858 28996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:14.941208 28996 raft_consensus.cc:504] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b665c098fb29424a80ad18b28a5b9daf" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 40013 } }
I20260928 10:49:14.941912 28996 leader_election.cc:302] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b665c098fb29424a80ad18b28a5b9daf; no voters:
I20260928 10:49:14.942580 28996 leader_election.cc:288] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:14.942700 29001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:14.943368 29001 raft_consensus.cc:686] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [term 1 LEADER]: Becoming Leader. State: Replica: b665c098fb29424a80ad18b28a5b9daf, State: Running, Role: LEADER
I20260928 10:49:14.943949 29001 consensus_queue.cc:237] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b665c098fb29424a80ad18b28a5b9daf" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 40013 } }
I20260928 10:49:14.945096 28996 sys_catalog.cc:564] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:14.947546 29002 sys_catalog.cc:455] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b665c098fb29424a80ad18b28a5b9daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b665c098fb29424a80ad18b28a5b9daf" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 40013 } } }
I20260928 10:49:14.948119 29002 sys_catalog.cc:458] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:14.948668 29003 sys_catalog.cc:455] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b665c098fb29424a80ad18b28a5b9daf. Latest consensus state: current_term: 1 leader_uuid: "b665c098fb29424a80ad18b28a5b9daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b665c098fb29424a80ad18b28a5b9daf" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 40013 } } }
I20260928 10:49:14.949023 29003 sys_catalog.cc:458] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:14.955696 29010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:14.960712 29010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:14.969645 29010 catalog_manager.cc:1347] Generated new cluster ID: 5e729e9adb3b47ca811daeda0a482442
I20260928 10:49:14.969782 29010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:14.982153 29010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:14.984182 29010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:14.992709 29010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf: Generated new TSK 0
I20260928 10:49:14.994031 29010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:14.998876 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.193:0
--local_ip_for_outbound_sockets=127.8.218.193
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:40013
--builtin_ntp_servers=127.8.218.212:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 10:49:15.357527 29020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:15.357849 29020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:15.357980 29020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:15.365559 29020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:15.365671 29020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:15.365814 29020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.193
I20260928 10:49:15.375634 29020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.218.193
--webserver_port=0
--tserver_master_addrs=127.8.218.254:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.193
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:15.376981 29020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:15.378772 29020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:15.392647 29028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:15.392763 29026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:15.392748 29025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:15.393782 29020 server_base.cc:1034] running on GCE node
I20260928 10:49:15.394402 29020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:15.395464 29020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:15.396740 29020 hybrid_clock.cc:648] HybridClock initialized: now 1790592555396685 us; error 44 us; skew 500 ppm
I20260928 10:49:15.397094 29020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:15.399644 29020 webserver.cc:460] Webserver started at http://127.8.218.193:45271/ using document root <none> and password file <none>
I20260928 10:49:15.400398 29020 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:15.400517 29020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:15.400838 29020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:15.404012 29020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/data/instance:
uuid: "03b270f9e6254181bbc8664910b1d6e7"
format_stamp: "Formatted at 2026-09-28 10:49:15 on dist-test-slave-mn38"
server_key: "b03df75f9e2d94357b6f75b11aabc45e"
server_key_iv: "1c11bc6960e0795f0a47de8711432c5e"
server_key_version: "encryptionkey@0"
I20260928 10:49:15.404838 29020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance:
uuid: "03b270f9e6254181bbc8664910b1d6e7"
format_stamp: "Formatted at 2026-09-28 10:49:15 on dist-test-slave-mn38"
server_key: "b03df75f9e2d94357b6f75b11aabc45e"
server_key_iv: "1c11bc6960e0795f0a47de8711432c5e"
server_key_version: "encryptionkey@0"
I20260928 10:49:15.410347 29020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 10:49:15.414378 29034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:15.415988 29020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:15.416188 29020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/wal
uuid: "03b270f9e6254181bbc8664910b1d6e7"
format_stamp: "Formatted at 2026-09-28 10:49:15 on dist-test-slave-mn38"
server_key: "b03df75f9e2d94357b6f75b11aabc45e"
server_key_iv: "1c11bc6960e0795f0a47de8711432c5e"
server_key_version: "encryptionkey@0"
I20260928 10:49:15.416428 29020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:15.436596 29020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:15.437569 29020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:15.437906 29020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:15.438957 29020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:15.440541 29020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:15.440675 29020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:15.440795 29020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:15.440862 29020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:15.472906 29020 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.193:39827
I20260928 10:49:15.472958 29146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.193:39827 every 8 connection(s)
I20260928 10:49:15.474278 29020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/data/info.pb
I20260928 10:49:15.474596 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 29020
I20260928 10:49:15.474756 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/wal/instance
I20260928 10:49:15.475286 9067 external_mini_cluster.cc:1428] Setting key 9a17dd75b407be1f51455f9b3081ee74
I20260928 10:49:15.478300 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.194:0
--local_ip_for_outbound_sockets=127.8.218.194
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:40013
--builtin_ntp_servers=127.8.218.212:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:49:15.488315 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:15.477134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.193:33423 (local address 127.8.218.254:40013)
0928 10:49:15.477628 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:15.477634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:15.479492 (+ 1858us) server_negotiation.cc:407] Connection header received
0928 10:49:15.481238 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:15.481243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:15.481322 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:15.481456 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:15.482575 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:15.483554 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:15.485457 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:15.485895 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:15.486020 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:15.487427 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:15.487433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:15.487436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:15.487457 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:15.487505 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:15.487519 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:15.487597 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:15.487836 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 10:49:15.487953 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":226,"threads_started":1}
I20260928 10:49:15.489430 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:15.477821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:40013 (local address 127.8.218.193:33423)
0928 10:49:15.479306 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:15.479356 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:15.480977 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:15.481676 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:15.481684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:15.481779 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:15.482399 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:15.482417 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:15.483717 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:15.483721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:15.485267 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:15.485282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:15.486144 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:15.486151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:15.486382 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:15.487070 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:15.487073 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:15.487117 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:15.487213 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:15.487611 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:15.487625 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:15.487633 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:49:15.487805 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 10:49:15.488266 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":761,"threads_started":1}
I20260928 10:49:15.491235 29147 heartbeater.cc:347] Connected to a master server at 127.8.218.254:40013
I20260928 10:49:15.491568 29147 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:15.492491 29147 heartbeater.cc:511] Master 127.8.218.254:40013 requested a full tablet report, sending...
I20260928 10:49:15.494935 28961 ts_manager.cc:194] Registered new tserver with Master: 03b270f9e6254181bbc8664910b1d6e7 (127.8.218.193:39827)
I20260928 10:49:15.496655 28961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.193:33423
W20260928 10:49:15.824398 29151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:15.824961 29151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:15.825124 29151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:15.833170 29151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:15.833352 29151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:15.833642 29151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.194
I20260928 10:49:15.843928 29151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.218.194
--webserver_port=0
--tserver_master_addrs=127.8.218.254:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.194
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:15.845294 29151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:15.847728 29151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:15.862353 29156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:15.862380 29157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:15.862526 29159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:15.863198 29151 server_base.cc:1034] running on GCE node
I20260928 10:49:15.863873 29151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:15.864897 29151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:15.866111 29151 hybrid_clock.cc:648] HybridClock initialized: now 1790592555866037 us; error 68 us; skew 500 ppm
I20260928 10:49:15.866441 29151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:15.868996 29151 webserver.cc:460] Webserver started at http://127.8.218.194:34707/ using document root <none> and password file <none>
I20260928 10:49:15.869809 29151 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:15.869920 29151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:15.870244 29151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:15.873276 29151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/data/instance:
uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b"
format_stamp: "Formatted at 2026-09-28 10:49:15 on dist-test-slave-mn38"
server_key: "af42dd2876a05260772ba6e67645d181"
server_key_iv: "95e17e0b991eb199d48f801d648f3c49"
server_key_version: "encryptionkey@0"
I20260928 10:49:15.874236 29151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance:
uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b"
format_stamp: "Formatted at 2026-09-28 10:49:15 on dist-test-slave-mn38"
server_key: "af42dd2876a05260772ba6e67645d181"
server_key_iv: "95e17e0b991eb199d48f801d648f3c49"
server_key_version: "encryptionkey@0"
I20260928 10:49:15.879974 29151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 10:49:15.883810 29165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:15.885241 29151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:49:15.885401 29151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/wal
uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b"
format_stamp: "Formatted at 2026-09-28 10:49:15 on dist-test-slave-mn38"
server_key: "af42dd2876a05260772ba6e67645d181"
server_key_iv: "95e17e0b991eb199d48f801d648f3c49"
server_key_version: "encryptionkey@0"
I20260928 10:49:15.885658 29151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:15.910262 29151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:15.911520 29151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:15.911896 29151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:15.913237 29151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:15.915431 29151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:15.915555 29151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:15.915671 29151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:15.915745 29151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:15.948364 29151 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.194:42285
I20260928 10:49:15.948457 29277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.194:42285 every 8 connection(s)
I20260928 10:49:15.949811 29151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/data/info.pb
I20260928 10:49:15.955746 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 29151
I20260928 10:49:15.955894 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/wal/instance
I20260928 10:49:15.956297 9067 external_mini_cluster.cc:1428] Setting key 8568f7025c8a784a5d018ccc5c6ffbab
I20260928 10:49:15.959515 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.218.195:0
--local_ip_for_outbound_sockets=127.8.218.195
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:40013
--builtin_ntp_servers=127.8.218.212:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 10:49:15.964308 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:15.952425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.194:42729 (local address 127.8.218.254:40013)
0928 10:49:15.952637 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:15.952641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:15.954136 (+ 1495us) server_negotiation.cc:407] Connection header received
0928 10:49:15.956263 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:15.956269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:15.956361 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:15.956458 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:15.957991 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:15.958902 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:15.961358 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:15.961807 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:15.961920 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:15.963483 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:15.963488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:15.963491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:15.963514 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:15.963553 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:15.963566 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:15.963649 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:15.963916 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 10:49:15.964014 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:49:15.965864 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:15.952959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:40013 (local address 127.8.218.194:42729)
0928 10:49:15.953958 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:15.953998 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:15.955717 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:15.956814 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:15.956828 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:15.956965 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:15.957807 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:15.957828 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:15.959093 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:15.959098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:15.961179 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:15.961198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:15.962042 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:15.962048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:15.962327 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:15.963087 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:15.963091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:15.963147 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:15.963261 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:15.963656 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:15.963665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:15.963673 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 10:49:15.963909 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 10:49:15.964379 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":270,"threads_started":1}
I20260928 10:49:15.967630 29278 heartbeater.cc:347] Connected to a master server at 127.8.218.254:40013
I20260928 10:49:15.967998 29278 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:15.968958 29278 heartbeater.cc:511] Master 127.8.218.254:40013 requested a full tablet report, sending...
I20260928 10:49:15.970840 28961 ts_manager.cc:194] Registered new tserver with Master: c8ce33d53ccc445c8faa4a5b4e021a6b (127.8.218.194:42285)
I20260928 10:49:15.971736 28961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.194:42729
W20260928 10:49:16.305522 29281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:16.305907 29281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:16.306037 29281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:16.314296 29281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:16.314443 29281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:16.314606 29281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.218.195
I20260928 10:49:16.324718 29281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.218.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.218.195
--webserver_port=0
--tserver_master_addrs=127.8.218.254:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.218.195
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:16.326217 29281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:16.328164 29281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:16.342546 29286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:16.342573 29287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:16.342519 29289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:16.343273 29281 server_base.cc:1034] running on GCE node
I20260928 10:49:16.344044 29281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:16.345178 29281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:16.346453 29281 hybrid_clock.cc:648] HybridClock initialized: now 1790592556346374 us; error 77 us; skew 500 ppm
I20260928 10:49:16.346814 29281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 10:49:16.349705 29281 webserver.cc:460] Webserver started at http://127.8.218.195:43555/ using document root <none> and password file <none>
I20260928 10:49:16.350535 29281 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:16.350669 29281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:16.351029 29281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:16.354703 29281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/data/instance:
uuid: "994118bb673c46b69745408e7466b3a2"
format_stamp: "Formatted at 2026-09-28 10:49:16 on dist-test-slave-mn38"
server_key: "65ac280fb2a98866b7ee6dd3fc790a3a"
server_key_iv: "f359f8d86480c815f15d1b9da8ee78d9"
server_key_version: "encryptionkey@0"
I20260928 10:49:16.355734 29281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance:
uuid: "994118bb673c46b69745408e7466b3a2"
format_stamp: "Formatted at 2026-09-28 10:49:16 on dist-test-slave-mn38"
server_key: "65ac280fb2a98866b7ee6dd3fc790a3a"
server_key_iv: "f359f8d86480c815f15d1b9da8ee78d9"
server_key_version: "encryptionkey@0"
I20260928 10:49:16.362476 29281 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 10:49:16.366117 29295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:16.367731 29281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:16.367923 29281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/wal
uuid: "994118bb673c46b69745408e7466b3a2"
format_stamp: "Formatted at 2026-09-28 10:49:16 on dist-test-slave-mn38"
server_key: "65ac280fb2a98866b7ee6dd3fc790a3a"
server_key_iv: "f359f8d86480c815f15d1b9da8ee78d9"
server_key_version: "encryptionkey@0"
I20260928 10:49:16.368129 29281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:16.379936 29281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:16.380751 29281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:16.381016 29281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:16.382045 29281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 10:49:16.383683 29281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 10:49:16.383769 29281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:16.383854 29281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 10:49:16.383908 29281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:16.416594 29281 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.195:45083
I20260928 10:49:16.416635 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.195:45083 every 8 connection(s)
I20260928 10:49:16.418151 29281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/data/info.pb
I20260928 10:49:16.419025 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 29281
I20260928 10:49:16.419195 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/wal/instance
I20260928 10:49:16.419842 9067 external_mini_cluster.cc:1428] Setting key 4f8602259883a24c9dc447f9d6532010
I20260928 10:49:16.434309 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.421577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:33763 (local address 127.8.218.254:40013)
0928 10:49:16.421800 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.421806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.423793 (+ 1987us) server_negotiation.cc:407] Connection header received
0928 10:49:16.425846 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.425851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.425947 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.426042 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:16.427481 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.428343 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.430574 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.431081 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.431205 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.432989 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:16.432996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:16.432999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:16.433058 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:16.433239 (+ 181us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:16.433274 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:16.433371 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.433848 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.433996 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:49:16.435648 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.422357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:40013 (local address 127.8.218.195:33763)
0928 10:49:16.423574 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:16.423628 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:16.425409 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:16.426406 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:16.426417 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:16.426550 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:16.427213 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.427235 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.428638 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.428643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.430328 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.430351 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.431369 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.431375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.431706 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.432501 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 10:49:16.432504 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:16.432573 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 10:49:16.432677 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:16.433486 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:16.433505 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:16.433516 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 10:49:16.433771 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 10:49:16.434273 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":359,"threads_started":1}
I20260928 10:49:16.437107 29408 heartbeater.cc:347] Connected to a master server at 127.8.218.254:40013
I20260928 10:49:16.437569 29408 heartbeater.cc:464] Registering TS with master...
I20260928 10:49:16.438517 29408 heartbeater.cc:511] Master 127.8.218.254:40013 requested a full tablet report, sending...
I20260928 10:49:16.440332 28961 ts_manager.cc:194] Registered new tserver with Master: 994118bb673c46b69745408e7466b3a2 (127.8.218.195:45083)
I20260928 10:49:16.441197 28961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.218.195:33763
I20260928 10:49:16.449329 9067 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 10:49:16.460043 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.453800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.8.218.254:40013)
0928 10:49:16.454056 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.454061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.454437 (+ 376us) server_negotiation.cc:407] Connection header received
0928 10:49:16.454574 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.454578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.454677 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.454767 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:16.455346 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.456165 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.457753 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.458246 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.458362 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.459470 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:16.459476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:16.459479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:16.459501 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:16.459537 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:16.459549 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:16.459619 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.459708 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.459791 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 10:49:16.464277 28961 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 10:49:16.471498 28961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 10:49:16.495951 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.484629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:42285 (local address 127.0.0.1:53256)
0928 10:49:16.487058 (+ 2429us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:16.487078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:16.487212 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:16.488169 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:16.488174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:16.488197 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:16.488624 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.488640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.490555 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.490564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.493347 (+ 2783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.493368 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.494890 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.494896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.495138 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.495196 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 10:49:16.495326 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 10:49:16.495552 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2262,"mutex_wait_us":363,"thread_start_us":1062,"threads_started":1}
I20260928 10:49:16.496304 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.485004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.8.218.194:42285)
0928 10:49:16.485733 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.485753 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.487347 (+ 1594us) server_negotiation.cc:407] Connection header received
0928 10:49:16.487455 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.487463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.487747 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.487930 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:16.488874 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.490372 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.493664 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.494670 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.494808 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.495112 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.495742 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.495944 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":405,"threads_started":1}
I20260928 10:49:16.496807 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.484721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.8.218.193:39827)
0928 10:49:16.486615 (+ 1894us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.486625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.486660 (+ 35us) server_negotiation.cc:407] Connection header received
0928 10:49:16.486832 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.486840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.487124 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.487326 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:16.488403 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.490262 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.493608 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.495171 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.495313 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.495608 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.496270 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.496452 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1719,"thread_start_us":1597,"threads_started":1}
I20260928 10:49:16.496146 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.484615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:39827 (local address 127.0.0.1:50304)
0928 10:49:16.486360 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:16.486399 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:16.486568 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:16.487658 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:16.487666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:16.487699 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:16.488151 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.488166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.490461 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.490467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.493345 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.493361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.495377 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.495383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.495553 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.495584 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 10:49:16.495692 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 10:49:16.495815 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"mutex_wait_us":112,"thread_start_us":2009,"threads_started":1}
I20260928 10:49:16.498893 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.487131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.195:45083 (local address 127.0.0.1:59504)
0928 10:49:16.489503 (+ 2372us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:16.489524 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:16.489663 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:16.490356 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:16.490361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:16.490390 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:16.491003 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.491045 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.492809 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.492815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.494769 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.494786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.498069 (+ 3283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.498076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.498254 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.498286 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 10:49:16.498377 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 10:49:16.498534 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2215,"mutex_wait_us":19,"thread_start_us":2064,"threads_started":1}
I20260928 10:49:16.498967 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.487567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.8.218.195:45083)
0928 10:49:16.489029 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.489039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.489641 (+ 602us) server_negotiation.cc:407] Connection header received
0928 10:49:16.489810 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.489819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.490114 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.490378 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:16.491238 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.492666 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.495045 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.497859 (+ 2814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.498002 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.498275 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.498443 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.498609 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1113,"threads_started":1}
I20260928 10:49:16.500252 29147 heartbeater.cc:503] Master 127.8.218.254:40013 was elected leader, sending a full tablet report...
I20260928 10:49:16.500869 29082 tablet_service.cc:1467] Processing CreateTablet for tablet 9401546b545a4213ae9f581ce1cb5e1b (DEFAULT_TABLE table=test-table [id=3df2c9bcecd54149b460ce186fd8c96c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:16.500852 29213 tablet_service.cc:1467] Processing CreateTablet for tablet 9401546b545a4213ae9f581ce1cb5e1b (DEFAULT_TABLE table=test-table [id=3df2c9bcecd54149b460ce186fd8c96c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:16.502497 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9401546b545a4213ae9f581ce1cb5e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:16.502502 29082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9401546b545a4213ae9f581ce1cb5e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:16.502935 29343 tablet_service.cc:1467] Processing CreateTablet for tablet 9401546b545a4213ae9f581ce1cb5e1b (DEFAULT_TABLE table=test-table [id=3df2c9bcecd54149b460ce186fd8c96c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 10:49:16.504302 29343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9401546b545a4213ae9f581ce1cb5e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:16.517117 29426 tablet_bootstrap.cc:492] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Bootstrap starting.
I20260928 10:49:16.520037 29428 tablet_bootstrap.cc:492] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Bootstrap starting.
I20260928 10:49:16.520576 29427 tablet_bootstrap.cc:492] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Bootstrap starting.
I20260928 10:49:16.522584 29426 tablet_bootstrap.cc:654] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:16.524132 29426 log.cc:824] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:16.524549 29428 tablet_bootstrap.cc:654] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:16.524565 29427 tablet_bootstrap.cc:654] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:16.525863 29427 log.cc:824] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:16.525938 29428 log.cc:824] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:16.528263 29426 tablet_bootstrap.cc:492] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: No bootstrap required, opened a new log
I20260928 10:49:16.528640 29426 ts_tablet_manager.cc:1397] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260928 10:49:16.529359 29428 tablet_bootstrap.cc:492] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: No bootstrap required, opened a new log
I20260928 10:49:16.529564 29427 tablet_bootstrap.cc:492] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: No bootstrap required, opened a new log
I20260928 10:49:16.529714 29428 ts_tablet_manager.cc:1397] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260928 10:49:16.529850 29427 ts_tablet_manager.cc:1397] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 10:49:16.537809 29427 raft_consensus.cc:348] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.538267 29427 raft_consensus.cc:374] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:16.538394 29427 raft_consensus.cc:729] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ce33d53ccc445c8faa4a5b4e021a6b, State: Initialized, Role: FOLLOWER
I20260928 10:49:16.539077 29427 consensus_queue.cc:260] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.540658 29428 raft_consensus.cc:348] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.541100 29427 ts_tablet_manager.cc:1428] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 10:49:16.541198 29278 heartbeater.cc:503] Master 127.8.218.254:40013 was elected leader, sending a full tablet report...
I20260928 10:49:16.541263 29428 raft_consensus.cc:374] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:16.541498 29428 raft_consensus.cc:729] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b270f9e6254181bbc8664910b1d6e7, State: Initialized, Role: FOLLOWER
I20260928 10:49:16.541368 29426 raft_consensus.cc:348] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.541977 29426 raft_consensus.cc:374] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:16.542116 29426 raft_consensus.cc:729] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994118bb673c46b69745408e7466b3a2, State: Initialized, Role: FOLLOWER
I20260928 10:49:16.542245 29428 consensus_queue.cc:260] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.542783 29426 consensus_queue.cc:260] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.543965 29408 heartbeater.cc:503] Master 127.8.218.254:40013 was elected leader, sending a full tablet report...
I20260928 10:49:16.545002 29426 ts_tablet_manager.cc:1428] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260928 10:49:16.546993 29428 ts_tablet_manager.cc:1428] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
W20260928 10:49:16.670409 29409 tablet.cc:2369] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:16.691758 29434 raft_consensus.cc:482] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 10:49:16.692091 29434 raft_consensus.cc:504] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.693876 29434 leader_election.cc:288] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03b270f9e6254181bbc8664910b1d6e7 (127.8.218.193:39827), c8ce33d53ccc445c8faa4a5b4e021a6b (127.8.218.194:42285)
I20260928 10:49:16.699747 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.694316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:40687 (local address 127.8.218.194:42285)
0928 10:49:16.694522 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.694528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.694552 (+ 24us) server_negotiation.cc:407] Connection header received
0928 10:49:16.694791 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.694798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.694897 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.695014 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:16.695458 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.696300 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.698098 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.698855 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.699029 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.699104 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.699353 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.699483 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:49:16.699787 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.694166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.194:42285 (local address 127.8.218.195:40687)
0928 10:49:16.694449 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:16.694471 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:16.694609 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:16.695107 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:16.695112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:16.695139 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:16.695294 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.695302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.696468 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.696471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.697925 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.697939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.699029 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.699035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.699198 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.699223 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 10:49:16.699302 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 10:49:16.699463 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260928 10:49:16.700274 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.694405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.218.195:58731 (local address 127.8.218.193:39827)
0928 10:49:16.694593 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.694599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.695182 (+ 583us) server_negotiation.cc:407] Connection header received
0928 10:49:16.695387 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.695392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.695482 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.695588 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 10:49:16.696106 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.696875 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.698632 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.699417 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.699575 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.699644 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.699930 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.700074 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 10:49:16.700948 29435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.694294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.193:39827 (local address 127.8.218.195:58731)
0928 10:49:16.695089 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:16.695108 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:16.695248 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:16.695629 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:16.695632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:16.695654 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 10:49:16.695948 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.695956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.697033 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.697038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.698439 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:16.698453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.699587 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.699593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:16.699753 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.699780 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 10:49:16.700461 (+ 681us) client_negotiation.cc:240] Negotiation successful
0928 10:49:16.700607 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"spinlock_wait_cycles":2944,"thread_start_us":439,"threads_started":1}
I20260928 10:49:16.700929 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9401546b545a4213ae9f581ce1cb5e1b" candidate_uuid: "994118bb673c46b69745408e7466b3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" is_pre_election: true
I20260928 10:49:16.701596 29233 raft_consensus.cc:2454] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 994118bb673c46b69745408e7466b3a2 in term 0.
W20260928 10:49:16.701881 29279 tablet.cc:2369] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:16.702157 29102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9401546b545a4213ae9f581ce1cb5e1b" candidate_uuid: "994118bb673c46b69745408e7466b3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03b270f9e6254181bbc8664910b1d6e7" is_pre_election: true
I20260928 10:49:16.702387 29299 leader_election.cc:302] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994118bb673c46b69745408e7466b3a2, c8ce33d53ccc445c8faa4a5b4e021a6b; no voters:
I20260928 10:49:16.702654 29102 raft_consensus.cc:2454] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 994118bb673c46b69745408e7466b3a2 in term 0.
I20260928 10:49:16.702831 29434 raft_consensus.cc:2781] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 10:49:16.702986 29434 raft_consensus.cc:482] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 10:49:16.703110 29434 raft_consensus.cc:3037] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:16.705344 29434 raft_consensus.cc:504] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.706007 29434 leader_election.cc:288] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [CANDIDATE]: Term 1 election: Requested vote from peers 03b270f9e6254181bbc8664910b1d6e7 (127.8.218.193:39827), c8ce33d53ccc445c8faa4a5b4e021a6b (127.8.218.194:42285)
I20260928 10:49:16.706552 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9401546b545a4213ae9f581ce1cb5e1b" candidate_uuid: "994118bb673c46b69745408e7466b3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b"
I20260928 10:49:16.706601 29102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9401546b545a4213ae9f581ce1cb5e1b" candidate_uuid: "994118bb673c46b69745408e7466b3a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03b270f9e6254181bbc8664910b1d6e7"
I20260928 10:49:16.706748 29233 raft_consensus.cc:3037] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:16.706853 29102 raft_consensus.cc:3037] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:16.709059 29233 raft_consensus.cc:2454] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994118bb673c46b69745408e7466b3a2 in term 1.
I20260928 10:49:16.709296 29102 raft_consensus.cc:2454] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994118bb673c46b69745408e7466b3a2 in term 1.
I20260928 10:49:16.709513 29299 leader_election.cc:302] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994118bb673c46b69745408e7466b3a2, c8ce33d53ccc445c8faa4a5b4e021a6b; no voters:
I20260928 10:49:16.709903 29434 raft_consensus.cc:2781] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:16.710352 29434 raft_consensus.cc:686] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 1 LEADER]: Becoming Leader. State: Replica: 994118bb673c46b69745408e7466b3a2, State: Running, Role: LEADER
I20260928 10:49:16.710817 29434 consensus_queue.cc:237] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } }
I20260928 10:49:16.715378 28961 catalog_manager.cc:5495] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 994118bb673c46b69745408e7466b3a2 (127.8.218.195). New cstate: current_term: 1 leader_uuid: "994118bb673c46b69745408e7466b3a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994118bb673c46b69745408e7466b3a2" member_type: VOTER last_known_addr { host: "127.8.218.195" port: 45083 } health_report { overall_health: HEALTHY } } }
W20260928 10:49:16.726783 29148 tablet.cc:2369] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:16.740868 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.735392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.8.218.195:45083)
0928 10:49:16.735609 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.735614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.735633 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:49:16.735875 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.735879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.735960 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.736118 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:16.736483 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.737392 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.738773 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.739214 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.739307 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.739737 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:16.740276 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:16.740393 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.740530 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.740600 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 10:49:16.749872 29233 raft_consensus.cc:1261] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 1 FOLLOWER]: Refusing update from remote peer 994118bb673c46b69745408e7466b3a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:16.749866 29102 raft_consensus.cc:1261] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 1 FOLLOWER]: Refusing update from remote peer 994118bb673c46b69745408e7466b3a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:16.751024 29436 consensus_queue.cc:1035] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b270f9e6254181bbc8664910b1d6e7" member_type: VOTER last_known_addr { host: "127.8.218.193" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:16.751446 29434 consensus_queue.cc:1035] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8ce33d53ccc445c8faa4a5b4e021a6b" member_type: VOTER last_known_addr { host: "127.8.218.194" port: 42285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:16.778895 28959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:16.779232 28959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 10:49:16.783432 28959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b665c098fb29424a80ad18b28a5b9daf: Sending DeleteTablet for 3 replicas of tablet 9401546b545a4213ae9f581ce1cb5e1b
I20260928 10:49:16.784814 29082 tablet_service.cc:1514] Processing DeleteTablet for tablet 9401546b545a4213ae9f581ce1cb5e1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:16 UTC) from {username='slave'} at 127.0.0.1:50304
I20260928 10:49:16.784838 29343 tablet_service.cc:1514] Processing DeleteTablet for tablet 9401546b545a4213ae9f581ce1cb5e1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:16 UTC) from {username='slave'} at 127.0.0.1:59504
I20260928 10:49:16.785660 29213 tablet_service.cc:1514] Processing DeleteTablet for tablet 9401546b545a4213ae9f581ce1cb5e1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 10:49:16 UTC) from {username='slave'} at 127.0.0.1:53256
I20260928 10:49:16.785917 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29020
I20260928 10:49:16.786042 29449 tablet_replica.cc:330] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: stopping tablet replica
I20260928 10:49:16.786484 29449 raft_consensus.cc:2229] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:16.792706 29449 raft_consensus.cc:2258] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:16.793741 29448 tablet_replica.cc:330] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: stopping tablet replica
I20260928 10:49:16.794313 29448 raft_consensus.cc:2229] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 10:49:16.795050 29448 raft_consensus.cc:2258] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:16.798141 29450 tablet_replica.cc:330] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: stopping tablet replica
I20260928 10:49:16.798789 29450 raft_consensus.cc:2229] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 10:49:16.799149 29449 ts_tablet_manager.cc:1905] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:16.799530 29450 raft_consensus.cc:2258] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 10:49:16.801930 29448 ts_tablet_manager.cc:1905] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:16.803557 29450 ts_tablet_manager.cc:1905] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 10:49:16.807408 29449 ts_tablet_manager.cc:1918] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:16.807690 29449 log.cc:1196] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-0/wal/wals/9401546b545a4213ae9f581ce1cb5e1b
I20260928 10:49:16.808290 29449 ts_tablet_manager.cc:1939] T 9401546b545a4213ae9f581ce1cb5e1b P 03b270f9e6254181bbc8664910b1d6e7: Deleting consensus metadata
I20260928 10:49:16.810210 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.788232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.8.218.193:39827)
0928 10:49:16.788426 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.788433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.795743 (+ 7310us) server_negotiation.cc:407] Connection header received
0928 10:49:16.796524 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.796532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.796660 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.796747 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:16.797978 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.799081 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.803882 (+ 4801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.804626 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.804755 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.808214 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:16.808229 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:16.808249 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:16.809091 (+ 842us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:16.809160 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:16.809190 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:16.809304 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.809674 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.809807 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 10:49:16.810200 29448 ts_tablet_manager.cc:1918] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:16.810389 29448 log.cc:1196] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-2/wal/wals/9401546b545a4213ae9f581ce1cb5e1b
I20260928 10:49:16.810686 28945 catalog_manager.cc:4841] TS 03b270f9e6254181bbc8664910b1d6e7 (127.8.218.193:39827): tablet 9401546b545a4213ae9f581ce1cb5e1b (table test-table [id=3df2c9bcecd54149b460ce186fd8c96c]) successfully deleted
I20260928 10:49:16.811576 29142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:16.811537 29448 ts_tablet_manager.cc:1939] T 9401546b545a4213ae9f581ce1cb5e1b P 994118bb673c46b69745408e7466b3a2: Deleting consensus metadata
I20260928 10:49:16.813828 28945 catalog_manager.cc:4841] TS 994118bb673c46b69745408e7466b3a2 (127.8.218.195:45083): tablet 9401546b545a4213ae9f581ce1cb5e1b (table test-table [id=3df2c9bcecd54149b460ce186fd8c96c]) successfully deleted
I20260928 10:49:16.817317 29450 ts_tablet_manager.cc:1918] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 10:49:16.817533 29450 log.cc:1196] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Deleting WAL directory at /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790592230404718-9067-0/minicluster-data/ts-1/wal/wals/9401546b545a4213ae9f581ce1cb5e1b
I20260928 10:49:16.818091 29450 ts_tablet_manager.cc:1939] T 9401546b545a4213ae9f581ce1cb5e1b P c8ce33d53ccc445c8faa4a5b4e021a6b: Deleting consensus metadata
I20260928 10:49:16.820125 28948 catalog_manager.cc:4841] TS c8ce33d53ccc445c8faa4a5b4e021a6b (127.8.218.194:42285): tablet 9401546b545a4213ae9f581ce1cb5e1b (table test-table [id=3df2c9bcecd54149b460ce186fd8c96c]) successfully deleted
I20260928 10:49:16.904179 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 29020
I20260928 10:49:16.919450 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29151
I20260928 10:49:16.926697 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:16.920260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.8.218.194:42285)
0928 10:49:16.920455 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:16.920461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:16.920481 (+ 20us) server_negotiation.cc:407] Connection header received
0928 10:49:16.920663 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:16.920670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:16.920791 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:16.920881 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:16.921329 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.922162 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.923420 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:16.923816 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:16.923926 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:16.925040 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:16.925052 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:16.925070 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:16.925856 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:16.925932 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:16.925958 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:16.926070 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:16.926207 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 10:49:16.926356 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 10:49:16.927536 29273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:17.018522 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 29151
I20260928 10:49:17.039789 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29281
I20260928 10:49:17.049118 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:17.040774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.8.218.195:45083)
0928 10:49:17.041103 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:17.041108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:17.041391 (+ 283us) server_negotiation.cc:407] Connection header received
0928 10:49:17.041652 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:17.041657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:17.041777 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:17.041935 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:17.042320 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:17.043465 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:17.044889 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:17.045684 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:17.045888 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:17.047135 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:17.047148 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:17.047165 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:17.048033 (+ 868us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:17.048197 (+ 164us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:17.048243 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:17.048393 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:17.048581 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 10:49:17.048700 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:49:17.049995 29403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:17.169160 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 29281
I20260928 10:49:17.186256 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 28930
I20260928 10:49:17.187325 28991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:17.298331 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 28930
2026-09-28T10:49:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2945 ms)
[----------] 6 tests from EncryptionPolicyTest (22504 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:17 dist-test-slave-mn38 krb5kdc[29458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:17 dist-test-slave-mn38 krb5kdc[29458](info): set up 2 sockets
Sep 28 10:49:17 dist-test-slave-mn38 krb5kdc[29458](info): commencing operation
krb5kdc: starting...
W20260928 10:49:18.342635 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:18 dist-test-slave-mn38 krb5kdc[29458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:18.599475 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:18.948760 29471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:18.949085 29471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:18.949203 29471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:18.956815 29471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:18.956977 29471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:18.957029 29471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:18.957116 29471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:18.957176 29471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:18.967558 29471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35209
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:18.968897 29471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:18.970350 29471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:18.982272 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:18.982318 29479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:18.982318 29476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:18.983579 29471 server_base.cc:1034] running on GCE node
I20260928 10:49:18.984515 29471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:18.986195 29471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:18.987434 29471 hybrid_clock.cc:648] HybridClock initialized: now 1790592558987387 us; error 40 us; skew 500 ppm
I20260928 10:49:18.987838 29471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:18 dist-test-slave-mn38 krb5kdc[29458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:18.991030 29471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:18.992693 29471 webserver.cc:460] Webserver started at http://127.0.0.1:34557/ using document root <none> and password file <none>
I20260928 10:49:18.993578 29471 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:18.993709 29471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:18.994038 29471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:18.997337 29471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "1bfc402592d04ea0b2f8833b251f2289"
format_stamp: "Formatted at 2026-09-28 10:49:18 on dist-test-slave-mn38"
server_key: "6e0c0e75d44807b1535dc1d5a47692cf"
server_key_iv: "5893a81e1a09d87bf5cf5b35583b0b4e"
server_key_version: "encryptionkey@0"
I20260928 10:49:18.998245 29471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "1bfc402592d04ea0b2f8833b251f2289"
format_stamp: "Formatted at 2026-09-28 10:49:18 on dist-test-slave-mn38"
server_key: "6e0c0e75d44807b1535dc1d5a47692cf"
server_key_iv: "5893a81e1a09d87bf5cf5b35583b0b4e"
server_key_version: "encryptionkey@0"
I20260928 10:49:19.004752 29471 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:49:19.008860 29486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:19.010459 29471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 10:49:19.010643 29471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "1bfc402592d04ea0b2f8833b251f2289"
format_stamp: "Formatted at 2026-09-28 10:49:18 on dist-test-slave-mn38"
server_key: "6e0c0e75d44807b1535dc1d5a47692cf"
server_key_iv: "5893a81e1a09d87bf5cf5b35583b0b4e"
server_key_version: "encryptionkey@0"
I20260928 10:49:19.010843 29471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:19.039511 29471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:19.044286 29471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:19.044571 29471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:19.062973 29471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35209
I20260928 10:49:19.062992 29537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35209 every 8 connection(s)
I20260928 10:49:19.064496 29471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:19.065135 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 29471
I20260928 10:49:19.065328 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:19.066076 9067 external_mini_cluster.cc:1428] Setting key 4426245ffe622d9b7977ebff8e5cb8e5
I20260928 10:49:19.069388 29538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 10:49:19 dist-test-slave-mn38 krb5kdc[29458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 10:49:19.083822 29538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289: Bootstrap starting.
I20260928 10:49:19.087738 29538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:19.089061 29538 log.cc:824] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:19.091624 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:19.068145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57779 (local address 127.0.0.1:35209)
0928 10:49:19.068978 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:19.068996 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:19.069053 (+ 57us) server_negotiation.cc:407] Connection header received
0928 10:49:19.071488 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:19.071524 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:19.072113 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:19.072645 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:19.074336 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:19.076476 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:19.077952 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:19.079030 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:19.079234 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:19.083660 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:19.083670 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:19.083687 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:19.083737 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:19.086945 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:19.088056 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:19.088064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:19.088070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:19.088292 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:19.088763 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:19.088766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:19.088768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:19.089354 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:19.089698 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:19.090251 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 10:49:19.090588 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":315,"threads_started":1}
I20260928 10:49:19.092234 29538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289: No bootstrap required, opened a new log
I20260928 10:49:19.095850 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:19.100716 29538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc402592d04ea0b2f8833b251f2289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } }
I20260928 10:49:19.101300 29538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:19.101424 29538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfc402592d04ea0b2f8833b251f2289, State: Initialized, Role: FOLLOWER
I20260928 10:49:19.102174 29538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc402592d04ea0b2f8833b251f2289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } }
I20260928 10:49:19.102459 29538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:19.102617 29538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:19.102823 29538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:19.105840 29538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc402592d04ea0b2f8833b251f2289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } }
I20260928 10:49:19.106554 29538 leader_election.cc:302] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bfc402592d04ea0b2f8833b251f2289; no voters:
I20260928 10:49:19.107235 29538 leader_election.cc:288] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:19.107342 29548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:19.108023 29548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [term 1 LEADER]: Becoming Leader. State: Replica: 1bfc402592d04ea0b2f8833b251f2289, State: Running, Role: LEADER
I20260928 10:49:19.108647 29548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc402592d04ea0b2f8833b251f2289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } }
I20260928 10:49:19.110759 29548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bfc402592d04ea0b2f8833b251f2289. Latest consensus state: current_term: 1 leader_uuid: "1bfc402592d04ea0b2f8833b251f2289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc402592d04ea0b2f8833b251f2289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } } }
I20260928 10:49:19.111186 29548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:19.111856 29550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bfc402592d04ea0b2f8833b251f2289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc402592d04ea0b2f8833b251f2289" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } } }
I20260928 10:49:19.112279 29550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:19.112475 29538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:19.114367 29552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:19.119910 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:19.100634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36113 (local address 127.0.0.1:35209)
0928 10:49:19.100842 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:19.100848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:19.101204 (+ 356us) server_negotiation.cc:407] Connection header received
0928 10:49:19.101430 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:19.101462 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:19.101541 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:19.101629 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:19.103507 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:19.104345 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:19.106138 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:19.106651 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:19.106784 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:19.110338 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:19.110346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:19.110352 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:19.110412 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:19.114503 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:19.115769 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:19.115776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:19.115780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:19.115993 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:19.116651 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:19.116657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:19.116661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:19.117150 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:19.117548 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:19.119186 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0928 10:49:19.119463 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 10:49:19.128279 29552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:19.141100 29552 catalog_manager.cc:1347] Generated new cluster ID: ecbf8b8592dc49eb955d8382c9023c64
I20260928 10:49:19.141242 29552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:19.162135 29552 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:19.164091 29552 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:19.177157 29552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bfc402592d04ea0b2f8833b251f2289: Generated new TSK 0
I20260928 10:49:19.178026 29552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:19.186923 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29471
I20260928 10:49:19.189241 29533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:19.282945 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 29471
W20260928 10:49:19.297854 29456 connection.cc:535] client connection to 127.0.0.1:35209 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35209): Connection reset by peer (error 104)
2026-09-28T10:49:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:19 dist-test-slave-mn38 krb5kdc[29570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:19 dist-test-slave-mn38 krb5kdc[29570](info): set up 2 sockets
Sep 28 10:49:19 dist-test-slave-mn38 krb5kdc[29570](info): commencing operation
krb5kdc: starting...
W20260928 10:49:20.326714 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:20 dist-test-slave-mn38 krb5kdc[29570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:20.590021 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:20.940027 29583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:20.940341 29583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:20.940436 29583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:20.947996 29583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:20.948130 29583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:20.948177 29583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:20.948238 29583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:20.948282 29583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:20.958690 29583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46543
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:20.960067 29583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:20.961421 29583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:20.973999 29588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:20.974203 29589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:20.974058 29591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:20.975426 29583 server_base.cc:1034] running on GCE node
I20260928 10:49:20.976217 29583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:20.977695 29583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:20.978960 29583 hybrid_clock.cc:648] HybridClock initialized: now 1790592560978860 us; error 31 us; skew 500 ppm
I20260928 10:49:20.979315 29583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:20 dist-test-slave-mn38 krb5kdc[29570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:20.982410 29583 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:20.983934 29583 webserver.cc:460] Webserver started at http://127.0.0.1:44813/ using document root <none> and password file <none>
I20260928 10:49:20.984665 29583 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:20.984781 29583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:20.985071 29583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:20.988232 29583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "838d7c8ba99a47df83a6ec11f7c7811b"
format_stamp: "Formatted at 2026-09-28 10:49:20 on dist-test-slave-mn38"
server_key: "42434be785eb725f3c0036250a4b9580"
server_key_iv: "ad626ae133c306091aa7c987425b5b86"
server_key_version: "encryptionkey@0"
I20260928 10:49:20.989094 29583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "838d7c8ba99a47df83a6ec11f7c7811b"
format_stamp: "Formatted at 2026-09-28 10:49:20 on dist-test-slave-mn38"
server_key: "42434be785eb725f3c0036250a4b9580"
server_key_iv: "ad626ae133c306091aa7c987425b5b86"
server_key_version: "encryptionkey@0"
I20260928 10:49:20.995498 29583 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 10:49:20.999670 29598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:21.001204 29583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:49:21.001417 29583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "838d7c8ba99a47df83a6ec11f7c7811b"
format_stamp: "Formatted at 2026-09-28 10:49:20 on dist-test-slave-mn38"
server_key: "42434be785eb725f3c0036250a4b9580"
server_key_iv: "ad626ae133c306091aa7c987425b5b86"
server_key_version: "encryptionkey@0"
I20260928 10:49:21.001691 29583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:21.016168 29583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:21.021145 29583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:21.021538 29583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:21.040926 29583 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46543
I20260928 10:49:21.040956 29649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46543 every 8 connection(s)
I20260928 10:49:21.042537 29583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:21.046041 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 29583
I20260928 10:49:21.046234 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:21.046782 9067 external_mini_cluster.cc:1428] Setting key 686961cdafc15875162a1c0f2061bfaa
I20260928 10:49:21.047341 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:21.063498 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:21.049018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33527 (local address 127.0.0.1:46543)
0928 10:49:21.050089 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:21.050111 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:21.050512 (+ 401us) server_negotiation.cc:407] Connection header received
0928 10:49:21.052608 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:21.052646 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:21.053346 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:21.053924 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:21.054564 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:21.056627 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:21.058120 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:21.059197 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:21.059397 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:21.060596 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:21.060608 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:21.060629 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:21.061418 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:21.061551 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:21.061590 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:21.061743 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:21.061958 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 10:49:21.062157 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":443,"threads_started":1}
I20260928 10:49:21.063815 29650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b: Bootstrap starting.
I20260928 10:49:21.067090 29650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:21.067456 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:21.068204 29650 log.cc:824] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:21.071350 29650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b: No bootstrap required, opened a new log
I20260928 10:49:21.078305 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:21.072082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46549 (local address 127.0.0.1:46543)
0928 10:49:21.072257 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:21.072263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:21.072608 (+ 345us) server_negotiation.cc:407] Connection header received
0928 10:49:21.072793 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:21.072797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:21.072861 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:21.072944 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:21.073557 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:21.074465 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:21.075755 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:21.076251 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:21.076368 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:21.077534 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:21.077540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:21.077543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:21.077570 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:21.077618 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:21.077637 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:21.077743 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:21.077957 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 10:49:21.078022 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 10:49:21.079592 29650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838d7c8ba99a47df83a6ec11f7c7811b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46543 } }
I20260928 10:49:21.080073 29650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:21.080165 29650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838d7c8ba99a47df83a6ec11f7c7811b, State: Initialized, Role: FOLLOWER
I20260928 10:49:21.080726 29650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838d7c8ba99a47df83a6ec11f7c7811b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46543 } }
I20260928 10:49:21.080960 29650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:21.081084 29650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:21.081241 29650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:21.083680 29650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838d7c8ba99a47df83a6ec11f7c7811b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46543 } }
I20260928 10:49:21.084298 29650 leader_election.cc:302] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 838d7c8ba99a47df83a6ec11f7c7811b; no voters:
I20260928 10:49:21.084860 29650 leader_election.cc:288] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:21.084975 29660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:21.085773 29660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [term 1 LEADER]: Becoming Leader. State: Replica: 838d7c8ba99a47df83a6ec11f7c7811b, State: Running, Role: LEADER
I20260928 10:49:21.086319 29660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838d7c8ba99a47df83a6ec11f7c7811b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46543 } }
I20260928 10:49:21.086886 29650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:21.089001 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838d7c8ba99a47df83a6ec11f7c7811b. Latest consensus state: current_term: 1 leader_uuid: "838d7c8ba99a47df83a6ec11f7c7811b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838d7c8ba99a47df83a6ec11f7c7811b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46543 } } }
I20260928 10:49:21.089392 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:21.090790 29661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838d7c8ba99a47df83a6ec11f7c7811b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838d7c8ba99a47df83a6ec11f7c7811b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46543 } } }
I20260928 10:49:21.091095 29661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:21.092008 29666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:21.097980 29666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:21.107687 29666 catalog_manager.cc:1347] Generated new cluster ID: 36dc20df640140d7ba453fd84fdd36c2
I20260928 10:49:21.107820 29666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:21.156530 29666 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:21.157956 29666 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:21.166522 29666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838d7c8ba99a47df83a6ec11f7c7811b: Generated new TSK 0
I20260928 10:49:21.167645 29666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:21.183938 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29583
W20260928 10:49:21.185204 29645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33527
W20260928 10:49:21.185952 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:21.186142 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 29583
W20260928 10:49:21.201663 29568 connection.cc:535] client connection to 127.0.0.1:46543 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46543): Connection reset by peer (error 104)
2026-09-28T10:49:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:21 dist-test-slave-mn38 krb5kdc[29681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:21 dist-test-slave-mn38 krb5kdc[29681](info): set up 2 sockets
Sep 28 10:49:21 dist-test-slave-mn38 krb5kdc[29681](info): commencing operation
krb5kdc: starting...
W20260928 10:49:22.226804 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:22 dist-test-slave-mn38 krb5kdc[29681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:22.505865 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:22.869076 29694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:22.869453 29694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:22.869592 29694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:22.877182 29694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:22.877323 29694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:22.877370 29694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:22.877457 29694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:22.877528 29694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:22.888182 29694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:22.889736 29694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:22.891180 29694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:22.904485 29699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:22.904613 29702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:22.904577 29700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:22.906113 29694 server_base.cc:1034] running on GCE node
I20260928 10:49:22.907155 29694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:22.908967 29694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:22.910298 29694 hybrid_clock.cc:648] HybridClock initialized: now 1790592562910229 us; error 65 us; skew 500 ppm
I20260928 10:49:22.910733 29694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:22 dist-test-slave-mn38 krb5kdc[29681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:22.917896 29694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:22.919818 29694 webserver.cc:460] Webserver started at http://127.0.0.1:44439/ using document root <none> and password file <none>
I20260928 10:49:22.920591 29694 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:22.920747 29694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:22.921072 29694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:22.924479 29694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "5ab82ffbc34540458a042a6941c34621"
format_stamp: "Formatted at 2026-09-28 10:49:22 on dist-test-slave-mn38"
server_key: "e0e9007af5f1e7633d34c599b2c096ed"
server_key_iv: "4f1ccd2898a653e10a26340de6381c24"
server_key_version: "encryptionkey@0"
I20260928 10:49:22.925324 29694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "5ab82ffbc34540458a042a6941c34621"
format_stamp: "Formatted at 2026-09-28 10:49:22 on dist-test-slave-mn38"
server_key: "e0e9007af5f1e7633d34c599b2c096ed"
server_key_iv: "4f1ccd2898a653e10a26340de6381c24"
server_key_version: "encryptionkey@0"
I20260928 10:49:22.931293 29694 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 10:49:22.935379 29709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:22.937083 29694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:49:22.937309 29694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "5ab82ffbc34540458a042a6941c34621"
format_stamp: "Formatted at 2026-09-28 10:49:22 on dist-test-slave-mn38"
server_key: "e0e9007af5f1e7633d34c599b2c096ed"
server_key_iv: "4f1ccd2898a653e10a26340de6381c24"
server_key_version: "encryptionkey@0"
I20260928 10:49:22.937604 29694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:22.944080 29694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:22.948777 29694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:22.949079 29694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:22.969532 29694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42781
I20260928 10:49:22.969590 29760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42781 every 8 connection(s)
I20260928 10:49:22.971388 29694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:22.973097 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 29694
I20260928 10:49:22.973652 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:22.974310 9067 external_mini_cluster.cc:1428] Setting key cac32a50dfdbcd49171eefb398eabcc7
I20260928 10:49:22.977278 29761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:22.986299 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:22.976575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37991 (local address 127.0.0.1:42781)
0928 10:49:22.977661 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:22.977677 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:22.979781 (+ 2104us) server_negotiation.cc:407] Connection header received
0928 10:49:22.981998 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:22.982031 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:22.982559 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:22.983056 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:22.984121 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:22.984131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:22.984149 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:22.984170 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:22.984260 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:22.984292 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:22.984534 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:22.984736 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 10:49:22.984960 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":421,"threads_started":1}
I20260928 10:49:22.990949 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:22.992484 29761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621: Bootstrap starting.
I20260928 10:49:22.996853 29761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:22.998394 29761 log.cc:824] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:22.999308 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:22.995860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45033 (local address 127.0.0.1:42781)
0928 10:49:22.996069 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:22.996075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:22.996689 (+ 614us) server_negotiation.cc:407] Connection header received
0928 10:49:22.996944 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:22.996950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:22.997036 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:22.997140 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:22.998401 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:22.998406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:22.998409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:22.998430 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:22.998480 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:22.998496 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:22.998630 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:22.998824 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 10:49:22.998885 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 10:49:23.002204 29761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621: No bootstrap required, opened a new log
I20260928 10:49:23.011098 29761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab82ffbc34540458a042a6941c34621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } }
I20260928 10:49:23.011564 29761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:23.011680 29761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab82ffbc34540458a042a6941c34621, State: Initialized, Role: FOLLOWER
I20260928 10:49:23.012408 29761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab82ffbc34540458a042a6941c34621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } }
I20260928 10:49:23.012737 29761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:23.012900 29761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:23.013105 29761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:23.015552 29761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab82ffbc34540458a042a6941c34621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } }
I20260928 10:49:23.016114 29761 leader_election.cc:302] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ab82ffbc34540458a042a6941c34621; no voters:
I20260928 10:49:23.016978 29761 leader_election.cc:288] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:23.017369 29771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:23.018448 29771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [term 1 LEADER]: Becoming Leader. State: Replica: 5ab82ffbc34540458a042a6941c34621, State: Running, Role: LEADER
I20260928 10:49:23.019176 29771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab82ffbc34540458a042a6941c34621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } }
I20260928 10:49:23.019796 29761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:23.022627 29773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ab82ffbc34540458a042a6941c34621. Latest consensus state: current_term: 1 leader_uuid: "5ab82ffbc34540458a042a6941c34621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab82ffbc34540458a042a6941c34621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } } }
I20260928 10:49:23.022716 29772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ab82ffbc34540458a042a6941c34621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab82ffbc34540458a042a6941c34621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } } }
I20260928 10:49:23.023075 29773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:23.023084 29772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:23.025020 29777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:23.031512 29777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:23.042060 29777 catalog_manager.cc:1347] Generated new cluster ID: 1258c30a8fc8444fa20d3e951e416c57
I20260928 10:49:23.042217 29777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:23.082345 29777 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:23.082825 29777 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:23.092197 29777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ab82ffbc34540458a042a6941c34621: Generated new TSK 0
I20260928 10:49:23.093261 29777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:23.112681 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29694
W20260928 10:49:23.113996 29756 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37991
W20260928 10:49:23.114814 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29694: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:23.114979 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 29694
2026-09-28T10:49:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:23 dist-test-slave-mn38 krb5kdc[29792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:23 dist-test-slave-mn38 krb5kdc[29792](info): set up 2 sockets
Sep 28 10:49:23 dist-test-slave-mn38 krb5kdc[29792](info): commencing operation
krb5kdc: starting...
W20260928 10:49:24.170710 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:24 dist-test-slave-mn38 krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:24.459805 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:24.851802 29805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:24.852139 29805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:24.852320 29805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:24.860431 29805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:24.860584 29805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:24.860632 29805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:24.860687 29805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:24.860729 29805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:24.871083 29805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:24.872364 29805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:24.873692 29805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:24.886209 29810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:24.886134 29813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:24.886168 29811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:24.887051 29805 server_base.cc:1034] running on GCE node
I20260928 10:49:24.887965 29805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:24.889631 29805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:24.890857 29805 hybrid_clock.cc:648] HybridClock initialized: now 1790592564890788 us; error 57 us; skew 500 ppm
I20260928 10:49:24.891228 29805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:24 dist-test-slave-mn38 krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:24.894582 29805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:24.896407 29805 webserver.cc:460] Webserver started at http://127.0.0.1:35653/ using document root <none> and password file <none>
I20260928 10:49:24.897199 29805 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:24.897333 29805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:24.897696 29805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:24.900898 29805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "13e85d0bb4d949c1850ad63c52622708"
format_stamp: "Formatted at 2026-09-28 10:49:24 on dist-test-slave-mn38"
server_key: "a475f59acec90a84195bdbf57805a84a"
server_key_iv: "2d3ee34ab048eca09971c91241747e60"
server_key_version: "encryptionkey@0"
I20260928 10:49:24.901748 29805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "13e85d0bb4d949c1850ad63c52622708"
format_stamp: "Formatted at 2026-09-28 10:49:24 on dist-test-slave-mn38"
server_key: "a475f59acec90a84195bdbf57805a84a"
server_key_iv: "2d3ee34ab048eca09971c91241747e60"
server_key_version: "encryptionkey@0"
I20260928 10:49:24.907660 29805 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 10:49:24.912179 29820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:24.913868 29805 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 10:49:24.914088 29805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "13e85d0bb4d949c1850ad63c52622708"
format_stamp: "Formatted at 2026-09-28 10:49:24 on dist-test-slave-mn38"
server_key: "a475f59acec90a84195bdbf57805a84a"
server_key_iv: "2d3ee34ab048eca09971c91241747e60"
server_key_version: "encryptionkey@0"
I20260928 10:49:24.914345 29805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:24.939070 29805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:24.944211 29805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:24.944504 29805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:24.963285 29805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42907
I20260928 10:49:24.963308 29871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42907 every 8 connection(s)
I20260928 10:49:24.964752 29805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:24.968685 29872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:24.969411 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 29805
I20260928 10:49:24.969616 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:24.970064 9067 external_mini_cluster.cc:1428] Setting key 8e5fdfb0e4e320ae3371f1df522f8260
I20260928 10:49:24.983209 29872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708: Bootstrap starting.
Sep 28 10:49:24 dist-test-slave-mn38 krb5kdc[29792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 10:49:24.986783 29872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:24.988281 29872 log.cc:824] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:24.991979 29872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708: No bootstrap required, opened a new log
I20260928 10:49:24.996039 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:24.971895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54915 (local address 127.0.0.1:42907)
0928 10:49:24.972717 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:24.972739 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:24.972792 (+ 53us) server_negotiation.cc:407] Connection header received
0928 10:49:24.975184 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:24.975218 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:24.975835 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:24.976321 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:24.977754 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:24.979743 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:24.981098 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:24.982199 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:24.982383 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:24.986622 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:24.986645 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:24.986673 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:24.986735 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:24.990471 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:24.991281 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:24.991292 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:24.991301 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:24.991538 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:24.991994 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:24.991999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:24.992002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:24.992760 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:24.993100 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:24.993627 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 10:49:24.994199 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":320,"threads_started":1}
I20260928 10:49:25.002213 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:25.004691 29872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e85d0bb4d949c1850ad63c52622708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42907 } }
I20260928 10:49:25.005852 29872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:25.006075 29872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e85d0bb4d949c1850ad63c52622708, State: Initialized, Role: FOLLOWER
I20260928 10:49:25.007813 29872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e85d0bb4d949c1850ad63c52622708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42907 } }
I20260928 10:49:25.008407 29872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:25.008635 29872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:25.008905 29872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:25.012326 29872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e85d0bb4d949c1850ad63c52622708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42907 } }
I20260928 10:49:25.012975 29872 leader_election.cc:302] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13e85d0bb4d949c1850ad63c52622708; no voters:
I20260928 10:49:25.013784 29872 leader_election.cc:288] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:25.013911 29882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:25.015089 29882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [term 1 LEADER]: Becoming Leader. State: Replica: 13e85d0bb4d949c1850ad63c52622708, State: Running, Role: LEADER
I20260928 10:49:25.015659 29882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e85d0bb4d949c1850ad63c52622708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42907 } }
I20260928 10:49:25.016222 29872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:25.018567 29884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13e85d0bb4d949c1850ad63c52622708. Latest consensus state: current_term: 1 leader_uuid: "13e85d0bb4d949c1850ad63c52622708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e85d0bb4d949c1850ad63c52622708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42907 } } }
I20260928 10:49:25.018991 29884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:25.019706 29883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13e85d0bb4d949c1850ad63c52622708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e85d0bb4d949c1850ad63c52622708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42907 } } }
I20260928 10:49:25.019999 29883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:25.021986 29888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:25.028738 29888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:25.029906 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:25.010613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35461 (local address 127.0.0.1:42907)
0928 10:49:25.010814 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:25.010819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:25.011125 (+ 306us) server_negotiation.cc:407] Connection header received
0928 10:49:25.011274 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:25.011279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:25.011361 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:25.011482 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:25.013115 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:25.014358 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:25.016123 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:25.016786 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:25.016934 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:25.020740 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:25.020748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:25.020755 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:25.020821 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:25.025368 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:25.026441 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:25.026446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:25.026450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:25.026659 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:25.027151 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:25.027156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:25.027159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:25.027652 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:25.027915 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:25.029220 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0928 10:49:25.029517 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:49:25.044090 29888 catalog_manager.cc:1347] Generated new cluster ID: 5c990daf12ee470d902611b354ba8904
I20260928 10:49:25.044230 29888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:25.062748 29888 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:25.064744 29888 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:25.079936 29888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13e85d0bb4d949c1850ad63c52622708: Generated new TSK 0
I20260928 10:49:25.080842 29888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:25.094141 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29805
I20260928 10:49:25.095310 29867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:25.189020 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 29805
2026-09-28T10:49:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:25 dist-test-slave-mn38 krb5kdc[29904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:25 dist-test-slave-mn38 krb5kdc[29904](info): set up 2 sockets
Sep 28 10:49:25 dist-test-slave-mn38 krb5kdc[29904](info): commencing operation
krb5kdc: starting...
W20260928 10:49:26.226522 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:26 dist-test-slave-mn38 krb5kdc[29904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:26.491057 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:26.850292 29917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:26.850625 29917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:26.850757 29917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:26.858428 29917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:26.858559 29917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:26.858639 29917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:26.858716 29917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:26.858772 29917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:26.869288 29917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:26.870707 29917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:26.872081 29917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:26.884492 29922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:26.884438 29923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:26.884480 29925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:26.885201 29917 server_base.cc:1034] running on GCE node
I20260928 10:49:26.886114 29917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:26.887707 29917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:26.888983 29917 hybrid_clock.cc:648] HybridClock initialized: now 1790592566888872 us; error 105 us; skew 500 ppm
I20260928 10:49:26.889425 29917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:26 dist-test-slave-mn38 krb5kdc[29904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:26.892849 29917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:26.894770 29917 webserver.cc:460] Webserver started at http://127.0.0.1:38079/ using document root <none> and password file <none>
I20260928 10:49:26.895524 29917 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:26.895653 29917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:26.895989 29917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:26.899228 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab"
format_stamp: "Formatted at 2026-09-28 10:49:26 on dist-test-slave-mn38"
server_key: "a8a94ff123624c8854d6280f91ee067b"
server_key_iv: "bbc69ba0fd060c4fca1f5d7e1d4ad563"
server_key_version: "encryptionkey@0"
I20260928 10:49:26.900027 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab"
format_stamp: "Formatted at 2026-09-28 10:49:26 on dist-test-slave-mn38"
server_key: "a8a94ff123624c8854d6280f91ee067b"
server_key_iv: "bbc69ba0fd060c4fca1f5d7e1d4ad563"
server_key_version: "encryptionkey@0"
I20260928 10:49:26.905644 29917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 10:49:26.909518 29932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:26.911089 29917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:26.911288 29917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab"
format_stamp: "Formatted at 2026-09-28 10:49:26 on dist-test-slave-mn38"
server_key: "a8a94ff123624c8854d6280f91ee067b"
server_key_iv: "bbc69ba0fd060c4fca1f5d7e1d4ad563"
server_key_version: "encryptionkey@0"
I20260928 10:49:26.911496 29917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:26.937932 29917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:26.942456 29917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:26.942744 29917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:26.961031 29917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40263
I20260928 10:49:26.961061 29983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40263 every 8 connection(s)
I20260928 10:49:26.962545 29917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:26.966475 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:26.968078 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 29917
I20260928 10:49:26.968241 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:26.968695 9067 external_mini_cluster.cc:1428] Setting key 828365db094866a27efc0225bbc42c51
I20260928 10:49:26.981122 29984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab: Bootstrap starting.
I20260928 10:49:26.983287 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:26.970728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35797 (local address 127.0.0.1:40263)
0928 10:49:26.971743 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:26.971757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:26.971808 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:49:26.973730 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:26.973768 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:26.974580 (+ 812us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:26.975130 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:26.975760 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:26.977096 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:26.978524 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:26.979328 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:26.979513 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:26.980831 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:26.980841 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:26.980858 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:26.981574 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:26.981645 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:26.981672 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:26.981798 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:26.982024 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 10:49:26.982296 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":452,"threads_started":1}
I20260928 10:49:26.984864 29984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:26.986236 29984 log.cc:824] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:26.987437 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:26.989596 29984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab: No bootstrap required, opened a new log
I20260928 10:49:26.999611 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:26.991908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39891 (local address 127.0.0.1:40263)
0928 10:49:26.992101 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:26.992108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:26.992335 (+ 227us) server_negotiation.cc:407] Connection header received
0928 10:49:26.992525 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:26.992530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:26.992620 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:26.992746 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:26.993233 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:26.994316 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:26.995727 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:26.996404 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:26.996559 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:26.998183 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:26.998190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:26.998196 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:26.998228 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:26.998288 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:26.998379 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:26.998499 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:26.999036 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 10:49:26.999152 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:49:26.999953 29984 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } }
I20260928 10:49:27.000506 29984 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:27.000630 29984 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc2f73c9c0e4469b0cf7aefe91a37ab, State: Initialized, Role: FOLLOWER
I20260928 10:49:27.001499 29984 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } }
I20260928 10:49:27.001776 29984 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:27.001912 29984 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:27.002071 29984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:27.004392 29984 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } }
I20260928 10:49:27.005262 29984 leader_election.cc:302] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdc2f73c9c0e4469b0cf7aefe91a37ab; no voters:
I20260928 10:49:27.006320 29984 leader_election.cc:288] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:27.006485 29994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:27.008059 29994 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [term 1 LEADER]: Becoming Leader. State: Replica: bdc2f73c9c0e4469b0cf7aefe91a37ab, State: Running, Role: LEADER
I20260928 10:49:27.008668 29994 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } }
I20260928 10:49:27.009088 29984 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:27.012290 29996 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdc2f73c9c0e4469b0cf7aefe91a37ab. Latest consensus state: current_term: 1 leader_uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } } }
I20260928 10:49:27.012333 29995 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc2f73c9c0e4469b0cf7aefe91a37ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } } }
I20260928 10:49:27.012794 29995 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:27.012794 29996 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:27.013762 30002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:27.020192 30002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:27.029284 30002 catalog_manager.cc:1347] Generated new cluster ID: d86113f8c9c044f4a1ed9a4346987195
I20260928 10:49:27.029456 30002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:27.059342 30002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:27.061537 30002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:27.077414 30002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdc2f73c9c0e4469b0cf7aefe91a37ab: Generated new TSK 0
I20260928 10:49:27.078533 30002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:27.095872 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29917
W20260928 10:49:27.097059 29979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35797
W20260928 10:49:27.097805 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 29917: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:27.097939 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 29917
2026-09-28T10:49:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:27 dist-test-slave-mn38 krb5kdc[30015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:27 dist-test-slave-mn38 krb5kdc[30015](info): set up 2 sockets
Sep 28 10:49:27 dist-test-slave-mn38 krb5kdc[30015](info): commencing operation
krb5kdc: starting...
W20260928 10:49:28.126672 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:28 dist-test-slave-mn38 krb5kdc[30015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:28.386693 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:28.747579 30028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:28.747944 30028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:28.748039 30028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:28.755918 30028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:28.756079 30028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:28.756125 30028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:28.756191 30028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:28.756237 30028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:28.767601 30028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:28.769070 30028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:28.770560 30028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:28.784024 30034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:28.784543 30036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:28.784605 30033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:28.784893 30028 server_base.cc:1034] running on GCE node
I20260928 10:49:28.785813 30028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:28.787501 30028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:28.788751 30028 hybrid_clock.cc:648] HybridClock initialized: now 1790592568788680 us; error 65 us; skew 500 ppm
I20260928 10:49:28.789244 30028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:28 dist-test-slave-mn38 krb5kdc[30015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:28.796542 30028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:28.798511 30028 webserver.cc:460] Webserver started at http://127.0.0.1:45833/ using document root <none> and password file <none>
I20260928 10:49:28.799401 30028 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:28.799556 30028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:28.799929 30028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:28.803637 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "788c3841df4c4942aeaf2425a94ed92e"
format_stamp: "Formatted at 2026-09-28 10:49:28 on dist-test-slave-mn38"
server_key: "414000aac0a1de5a60b1fc24af4e6333"
server_key_iv: "55def9c0cac2a71331450c32a205b229"
server_key_version: "encryptionkey@0"
I20260928 10:49:28.804571 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "788c3841df4c4942aeaf2425a94ed92e"
format_stamp: "Formatted at 2026-09-28 10:49:28 on dist-test-slave-mn38"
server_key: "414000aac0a1de5a60b1fc24af4e6333"
server_key_iv: "55def9c0cac2a71331450c32a205b229"
server_key_version: "encryptionkey@0"
I20260928 10:49:28.810720 30028 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:49:28.814832 30043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:28.816480 30028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 10:49:28.816684 30028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "788c3841df4c4942aeaf2425a94ed92e"
format_stamp: "Formatted at 2026-09-28 10:49:28 on dist-test-slave-mn38"
server_key: "414000aac0a1de5a60b1fc24af4e6333"
server_key_iv: "55def9c0cac2a71331450c32a205b229"
server_key_version: "encryptionkey@0"
I20260928 10:49:28.816900 30028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:28.823678 30028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:28.828729 30028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:28.829070 30028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:28.851831 30028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37391
I20260928 10:49:28.851867 30094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37391 every 8 connection(s)
I20260928 10:49:28.853659 30028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:28.858774 30095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:28.863941 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30028
I20260928 10:49:28.864153 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:28.864709 9067 external_mini_cluster.cc:1428] Setting key 6b6a2a80ea8bf4704a9bd60e85644919
I20260928 10:49:28.873167 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:28.866900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59837 (local address 127.0.0.1:37391)
0928 10:49:28.867789 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:28.867802 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:28.867849 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:49:28.869656 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:28.869682 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:28.870115 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:28.870483 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:28.871525 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:28.871532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:28.871546 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:28.871561 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:28.871623 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:28.871644 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:28.871870 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:28.871977 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 10:49:28.872182 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":430,"threads_started":1}
I20260928 10:49:28.873793 30095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e: Bootstrap starting.
I20260928 10:49:28.876894 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:28.877130 30095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:28.878423 30095 log.cc:824] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:28.881984 30095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e: No bootstrap required, opened a new log
I20260928 10:49:28.884461 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:28.881894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54681 (local address 127.0.0.1:37391)
0928 10:49:28.882091 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:28.882098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:28.882490 (+ 392us) server_negotiation.cc:407] Connection header received
0928 10:49:28.882638 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:28.882643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:28.882710 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:28.882828 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:28.883756 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:28.883762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:28.883767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:28.883787 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:28.883845 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:28.883870 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:28.884007 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:28.884079 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 10:49:28.884143 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:49:28.891088 30095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c3841df4c4942aeaf2425a94ed92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } }
I20260928 10:49:28.891551 30095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:28.891657 30095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788c3841df4c4942aeaf2425a94ed92e, State: Initialized, Role: FOLLOWER
I20260928 10:49:28.892660 30095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c3841df4c4942aeaf2425a94ed92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } }
I20260928 10:49:28.893159 30095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:28.893482 30095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:28.893728 30095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:28.896176 30095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c3841df4c4942aeaf2425a94ed92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } }
I20260928 10:49:28.896749 30095 leader_election.cc:302] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788c3841df4c4942aeaf2425a94ed92e; no voters:
I20260928 10:49:28.897590 30095 leader_election.cc:288] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:28.897809 30105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:28.898594 30105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [term 1 LEADER]: Becoming Leader. State: Replica: 788c3841df4c4942aeaf2425a94ed92e, State: Running, Role: LEADER
I20260928 10:49:28.899158 30105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c3841df4c4942aeaf2425a94ed92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } }
I20260928 10:49:28.899900 30095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:28.902768 30106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788c3841df4c4942aeaf2425a94ed92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c3841df4c4942aeaf2425a94ed92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } } }
I20260928 10:49:28.903250 30106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:28.902994 30107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788c3841df4c4942aeaf2425a94ed92e. Latest consensus state: current_term: 1 leader_uuid: "788c3841df4c4942aeaf2425a94ed92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c3841df4c4942aeaf2425a94ed92e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } } }
I20260928 10:49:28.903371 30107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:28.905722 30112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:28.911213 30112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:28.921087 30112 catalog_manager.cc:1347] Generated new cluster ID: 6ee3e85ea2d045c28ff23d1d4907724b
I20260928 10:49:28.921258 30112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:28.945935 30112 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:28.946513 30112 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:28.956995 30112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788c3841df4c4942aeaf2425a94ed92e: Generated new TSK 0
I20260928 10:49:28.958148 30112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:28.967854 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30028
W20260928 10:49:28.969344 30090 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59837
W20260928 10:49:28.970218 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30028: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:28.970397 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30028
2026-09-28T10:49:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:29 dist-test-slave-mn38 krb5kdc[30126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:29 dist-test-slave-mn38 krb5kdc[30126](info): set up 2 sockets
Sep 28 10:49:29 dist-test-slave-mn38 krb5kdc[30126](info): commencing operation
krb5kdc: starting...
W20260928 10:49:30.007115 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:30 dist-test-slave-mn38 krb5kdc[30126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:30.279275 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:30.640113 30139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:30.640466 30139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:30.640568 30139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:30.649005 30139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:30.649168 30139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:30.649240 30139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:30.649333 30139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:30.649385 30139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:30.659969 30139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:30.661319 30139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:30.662708 30139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:30.674597 30145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:30.674621 30147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:30.674558 30144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:30.675475 30139 server_base.cc:1034] running on GCE node
I20260928 10:49:30.676311 30139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:30.677870 30139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:30.679075 30139 hybrid_clock.cc:648] HybridClock initialized: now 1790592570679025 us; error 43 us; skew 500 ppm
I20260928 10:49:30.679440 30139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:30 dist-test-slave-mn38 krb5kdc[30126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:30.683049 30139 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:30.684823 30139 webserver.cc:460] Webserver started at http://127.0.0.1:41559/ using document root <none> and password file <none>
I20260928 10:49:30.685642 30139 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:30.685766 30139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:30.686085 30139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:30.689779 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "47b722235ae5468ea0c0e13f934de6d5"
format_stamp: "Formatted at 2026-09-28 10:49:30 on dist-test-slave-mn38"
server_key: "f2aaa1d3d5e687ac4f0d62a9af772d1f"
server_key_iv: "7d84312b5615164fe994e838de943fc5"
server_key_version: "encryptionkey@0"
I20260928 10:49:30.690703 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "47b722235ae5468ea0c0e13f934de6d5"
format_stamp: "Formatted at 2026-09-28 10:49:30 on dist-test-slave-mn38"
server_key: "f2aaa1d3d5e687ac4f0d62a9af772d1f"
server_key_iv: "7d84312b5615164fe994e838de943fc5"
server_key_version: "encryptionkey@0"
I20260928 10:49:30.696985 30139 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 10:49:30.701069 30154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:30.702648 30139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:49:30.702894 30139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "47b722235ae5468ea0c0e13f934de6d5"
format_stamp: "Formatted at 2026-09-28 10:49:30 on dist-test-slave-mn38"
server_key: "f2aaa1d3d5e687ac4f0d62a9af772d1f"
server_key_iv: "7d84312b5615164fe994e838de943fc5"
server_key_version: "encryptionkey@0"
I20260928 10:49:30.703107 30139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:30.719261 30139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:30.724934 30139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:30.725548 30139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:30.745323 30139 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35755
I20260928 10:49:30.745326 30205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35755 every 8 connection(s)
I20260928 10:49:30.746990 30139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:30.751251 30206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:30.755772 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30139
I20260928 10:49:30.755942 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:30.756361 9067 external_mini_cluster.cc:1428] Setting key d8808bf9ffccad8665274883855d0735
I20260928 10:49:30.764081 30206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5: Bootstrap starting.
I20260928 10:49:30.767859 30206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:30.769158 30206 log.cc:824] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:30.772373 30206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5: No bootstrap required, opened a new log
Sep 28 10:49:30 dist-test-slave-mn38 krb5kdc[30126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 10:49:30.782671 30206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b722235ae5468ea0c0e13f934de6d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260928 10:49:30.783174 30206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:30.783295 30206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b722235ae5468ea0c0e13f934de6d5, State: Initialized, Role: FOLLOWER
I20260928 10:49:30.783354 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:30.758359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:37459 (local address 127.0.0.1:35755)
0928 10:49:30.759154 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:30.759171 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:30.759222 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:49:30.762279 (+ 3057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:30.762316 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:30.762912 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:30.763379 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:30.764844 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:30.766902 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:30.768285 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:30.769374 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:30.769606 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:30.774088 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:30.774099 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:30.774120 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:30.774178 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:30.778314 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:30.779271 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:30.779281 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:30.779291 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:30.779533 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:30.779964 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:30.779969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:30.779973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:30.780775 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:30.781099 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:30.781609 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 10:49:30.781967 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":325,"threads_started":1}
I20260928 10:49:30.784003 30206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b722235ae5468ea0c0e13f934de6d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260928 10:49:30.784260 30206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:30.784389 30206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:30.784552 30206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:30.787864 30206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b722235ae5468ea0c0e13f934de6d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260928 10:49:30.788655 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:30.788573 30206 leader_election.cc:302] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47b722235ae5468ea0c0e13f934de6d5; no voters:
I20260928 10:49:30.791085 30206 leader_election.cc:288] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:30.791241 30211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:30.798808 30211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [term 1 LEADER]: Becoming Leader. State: Replica: 47b722235ae5468ea0c0e13f934de6d5, State: Running, Role: LEADER
I20260928 10:49:30.799619 30211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b722235ae5468ea0c0e13f934de6d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260928 10:49:30.803592 30206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:30.811730 30213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47b722235ae5468ea0c0e13f934de6d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b722235ae5468ea0c0e13f934de6d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } } }
I20260928 10:49:30.812285 30213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:30.814708 30212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b722235ae5468ea0c0e13f934de6d5. Latest consensus state: current_term: 1 leader_uuid: "47b722235ae5468ea0c0e13f934de6d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b722235ae5468ea0c0e13f934de6d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } } }
I20260928 10:49:30.815042 30226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:30.815094 30212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:30.821261 30226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:30.832605 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:30.804021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:50575 (local address 127.0.0.1:35755)
0928 10:49:30.804256 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:30.804264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:30.811283 (+ 7019us) server_negotiation.cc:407] Connection header received
0928 10:49:30.811498 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:30.811506 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:30.811616 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:30.811748 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:30.814494 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:30.815878 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:30.818311 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:30.819218 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:30.819353 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:30.823398 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:30.823407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:30.823413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:30.823474 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:30.827733 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:30.829045 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:30.829053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:30.829058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:30.829274 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:30.830045 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:30.830052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:30.830056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:30.830595 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:30.831023 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:30.831871 (+ 848us) server_negotiation.cc:299] Negotiation successful
0928 10:49:30.832153 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:49:30.833992 30226 catalog_manager.cc:1347] Generated new cluster ID: 816b6f328bde47bcb86a08629e18f1f4
I20260928 10:49:30.834129 30226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:30.849972 30226 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:30.851572 30226 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:30.870457 30226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47b722235ae5468ea0c0e13f934de6d5: Generated new TSK 0
I20260928 10:49:30.871333 30226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:30.889077 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30139
I20260928 10:49:30.890659 30201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:30.986479 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30139
W20260928 10:49:31.001910 30124 connection.cc:535] client connection to 127.0.0.1:35755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35755): Connection reset by peer (error 104)
2026-09-28T10:49:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:31 dist-test-slave-mn38 krb5kdc[30239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:31 dist-test-slave-mn38 krb5kdc[30239](info): set up 2 sockets
Sep 28 10:49:31 dist-test-slave-mn38 krb5kdc[30239](info): commencing operation
krb5kdc: starting...
W20260928 10:49:32.026772 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:32 dist-test-slave-mn38 krb5kdc[30239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:32.292830 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:32.661123 30252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:32.661479 30252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:32.661633 30252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:32.669114 30252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:32.669255 30252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:32.669349 30252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:32.669428 30252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:32.669507 30252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:32.680145 30252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:32.681861 30252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:32.683285 30252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:32.695959 30257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:32.695907 30258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:32.695962 30260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:32.697405 30252 server_base.cc:1034] running on GCE node
I20260928 10:49:32.698330 30252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:32.699872 30252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:32.701089 30252 hybrid_clock.cc:648] HybridClock initialized: now 1790592572701046 us; error 37 us; skew 500 ppm
I20260928 10:49:32.701499 30252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:32 dist-test-slave-mn38 krb5kdc[30239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:32.704823 30252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:32.706489 30252 webserver.cc:460] Webserver started at http://127.0.0.1:43889/ using document root <none> and password file <none>
I20260928 10:49:32.707211 30252 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:32.707394 30252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:32.707726 30252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:32.711014 30252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "2122576d077c449b8e3a03ce324f3d3a"
format_stamp: "Formatted at 2026-09-28 10:49:32 on dist-test-slave-mn38"
server_key: "6ae9a335b77b54269a62d35bd0450ce9"
server_key_iv: "38ae4ea5bdd51a45bb4bc12de4141c18"
server_key_version: "encryptionkey@0"
I20260928 10:49:32.711879 30252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "2122576d077c449b8e3a03ce324f3d3a"
format_stamp: "Formatted at 2026-09-28 10:49:32 on dist-test-slave-mn38"
server_key: "6ae9a335b77b54269a62d35bd0450ce9"
server_key_iv: "38ae4ea5bdd51a45bb4bc12de4141c18"
server_key_version: "encryptionkey@0"
I20260928 10:49:32.717778 30252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 10:49:32.721763 30267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:32.723397 30252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:32.723606 30252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "2122576d077c449b8e3a03ce324f3d3a"
format_stamp: "Formatted at 2026-09-28 10:49:32 on dist-test-slave-mn38"
server_key: "6ae9a335b77b54269a62d35bd0450ce9"
server_key_iv: "38ae4ea5bdd51a45bb4bc12de4141c18"
server_key_version: "encryptionkey@0"
I20260928 10:49:32.723812 30252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:32.753721 30252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:32.758536 30252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:32.758863 30252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:32.778086 30252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33423
I20260928 10:49:32.778122 30318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33423 every 8 connection(s)
I20260928 10:49:32.779470 30252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:32.780934 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30252
I20260928 10:49:32.781083 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:32.781522 9067 external_mini_cluster.cc:1428] Setting key 40c3891f9d517e0cb048f971fa6f26c3
I20260928 10:49:32.784008 30319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:32.798563 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:32.783358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:58587 (local address 127.0.0.1:33423)
0928 10:49:32.784165 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:32.784181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:32.784234 (+ 53us) server_negotiation.cc:407] Connection header received
0928 10:49:32.786593 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:32.786625 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:32.787234 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:32.787751 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:32.788353 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:32.790534 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:32.792545 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:32.793759 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:32.793986 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:32.795456 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:32.795469 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:32.795497 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:32.796407 (+ 910us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:32.796514 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:32.796549 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:32.796703 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:32.797068 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 10:49:32.797275 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":315,"threads_started":1}
I20260928 10:49:32.799635 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a: Bootstrap starting.
I20260928 10:49:32.802584 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:32.803099 30319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:32.804098 30319 log.cc:824] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:32.807265 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a: No bootstrap required, opened a new log
I20260928 10:49:32.814189 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:32.806935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:36513 (local address 127.0.0.1:33423)
0928 10:49:32.807131 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:32.807137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:32.807421 (+ 284us) server_negotiation.cc:407] Connection header received
0928 10:49:32.807621 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:32.807627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:32.807724 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:32.807858 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:32.808317 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:32.809526 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:32.811054 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:32.811759 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:32.811908 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:32.813152 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:32.813159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:32.813164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:32.813193 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:32.813252 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:32.813273 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:32.813388 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:32.813764 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 10:49:32.813833 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 10:49:32.817512 30319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122576d077c449b8e3a03ce324f3d3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } }
I20260928 10:49:32.817957 30319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:32.818063 30319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2122576d077c449b8e3a03ce324f3d3a, State: Initialized, Role: FOLLOWER
I20260928 10:49:32.818686 30319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122576d077c449b8e3a03ce324f3d3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } }
I20260928 10:49:32.818969 30319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:32.819123 30319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:32.819293 30319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:32.821887 30319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122576d077c449b8e3a03ce324f3d3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } }
I20260928 10:49:32.822492 30319 leader_election.cc:302] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2122576d077c449b8e3a03ce324f3d3a; no voters:
I20260928 10:49:32.823109 30319 leader_election.cc:288] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:32.823262 30329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:32.823954 30329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [term 1 LEADER]: Becoming Leader. State: Replica: 2122576d077c449b8e3a03ce324f3d3a, State: Running, Role: LEADER
I20260928 10:49:32.824584 30329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122576d077c449b8e3a03ce324f3d3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } }
I20260928 10:49:32.825291 30319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:32.827651 30331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2122576d077c449b8e3a03ce324f3d3a. Latest consensus state: current_term: 1 leader_uuid: "2122576d077c449b8e3a03ce324f3d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122576d077c449b8e3a03ce324f3d3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } } }
I20260928 10:49:32.828024 30331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:32.832990 30330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2122576d077c449b8e3a03ce324f3d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2122576d077c449b8e3a03ce324f3d3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } } }
I20260928 10:49:32.833288 30330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:32.834316 30335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:32.839083 30335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:32.847899 30335 catalog_manager.cc:1347] Generated new cluster ID: de35d7d27e6747388bb5723b8a6fd448
I20260928 10:49:32.848098 30335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:32.883121 30335 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:32.885190 30335 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:32.898336 30335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2122576d077c449b8e3a03ce324f3d3a: Generated new TSK 0
I20260928 10:49:32.899554 30335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:32.905767 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30252
W20260928 10:49:32.907018 30314 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.39:58587
W20260928 10:49:32.907960 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:32.908135 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30252
W20260928 10:49:32.923224 30237 connection.cc:535] client connection to 127.0.0.1:33423 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33423): Connection reset by peer (error 104)
2026-09-28T10:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:33 dist-test-slave-mn38 krb5kdc[30350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:33 dist-test-slave-mn38 krb5kdc[30350](info): set up 2 sockets
Sep 28 10:49:33 dist-test-slave-mn38 krb5kdc[30350](info): commencing operation
krb5kdc: starting...
W20260928 10:49:33.938851 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:34 dist-test-slave-mn38 krb5kdc[30350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:34.211088 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:34.560490 30363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:34.560834 30363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:34.560931 30363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:34.568713 30363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:34.568846 30363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:34.568893 30363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:34.568953 30363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:34.568995 30363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:34.579298 30363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:34.580570 30363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:34.581881 30363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:34.594038 30371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:34.594067 30369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:34.594028 30368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:34.594595 30363 server_base.cc:1034] running on GCE node
I20260928 10:49:34.595709 30363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:34.597332 30363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:34.598562 30363 hybrid_clock.cc:648] HybridClock initialized: now 1790592574598492 us; error 72 us; skew 500 ppm
I20260928 10:49:34.598927 30363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:34 dist-test-slave-mn38 krb5kdc[30350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:34.607913 30363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:34.609758 30363 webserver.cc:460] Webserver started at http://127.0.0.1:33701/ using document root <none> and password file <none>
I20260928 10:49:34.610544 30363 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:34.610679 30363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:34.611008 30363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:34.614274 30363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "8b5376da67144133907d832e0847782f"
format_stamp: "Formatted at 2026-09-28 10:49:34 on dist-test-slave-mn38"
server_key: "5a92fdc757152995e1e441d906d5a301"
server_key_iv: "c6a133786f185fd39f8959ccf66ba31e"
server_key_version: "encryptionkey@0"
I20260928 10:49:34.615103 30363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "8b5376da67144133907d832e0847782f"
format_stamp: "Formatted at 2026-09-28 10:49:34 on dist-test-slave-mn38"
server_key: "5a92fdc757152995e1e441d906d5a301"
server_key_iv: "c6a133786f185fd39f8959ccf66ba31e"
server_key_version: "encryptionkey@0"
I20260928 10:49:34.621091 30363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260928 10:49:34.625679 30378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:34.627382 30363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:49:34.627604 30363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "8b5376da67144133907d832e0847782f"
format_stamp: "Formatted at 2026-09-28 10:49:34 on dist-test-slave-mn38"
server_key: "5a92fdc757152995e1e441d906d5a301"
server_key_iv: "c6a133786f185fd39f8959ccf66ba31e"
server_key_version: "encryptionkey@0"
I20260928 10:49:34.627842 30363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:34.634680 30363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:34.639911 30363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:34.640250 30363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:34.658963 30363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45113
I20260928 10:49:34.659019 30429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45113 every 8 connection(s)
I20260928 10:49:34.660367 30363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:34.664361 30430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:34.668013 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30363
I20260928 10:49:34.668198 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:34.668730 9067 external_mini_cluster.cc:1428] Setting key 70b8d7ed7d3f03bfcbce6bf32cff892b
I20260928 10:49:34.676594 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:34.670674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:34473 (local address 127.0.0.1:45113)
0928 10:49:34.671440 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:34.671453 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:34.671497 (+ 44us) server_negotiation.cc:407] Connection header received
0928 10:49:34.673164 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:34.673191 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:34.673602 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:34.673989 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:34.675016 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:34.675023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:34.675040 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:34.675055 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:34.675117 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:34.675140 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:34.675340 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:34.675471 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 10:49:34.675660 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":308,"threads_started":1}
I20260928 10:49:34.676966 30430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f: Bootstrap starting.
I20260928 10:49:34.680320 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:34.680404 30430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:34.681463 30430 log.cc:824] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:34.684258 30430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f: No bootstrap required, opened a new log
I20260928 10:49:34.687166 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:34.684410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:43167 (local address 127.0.0.1:45113)
0928 10:49:34.684579 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:34.684585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:34.685043 (+ 458us) server_negotiation.cc:407] Connection header received
0928 10:49:34.685170 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:34.685175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:34.685238 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:34.685355 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:34.686407 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:34.686413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:34.686418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:34.686437 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:34.686494 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:34.686514 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:34.686660 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:34.686799 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 10:49:34.686868 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 10:49:34.693406 30430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5376da67144133907d832e0847782f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45113 } }
I20260928 10:49:34.693914 30430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:34.693996 30430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5376da67144133907d832e0847782f, State: Initialized, Role: FOLLOWER
I20260928 10:49:34.694777 30430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5376da67144133907d832e0847782f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45113 } }
I20260928 10:49:34.695111 30430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:34.695300 30430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:34.695528 30430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:34.698009 30430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5376da67144133907d832e0847782f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45113 } }
I20260928 10:49:34.698553 30430 leader_election.cc:302] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b5376da67144133907d832e0847782f; no voters:
I20260928 10:49:34.699180 30430 leader_election.cc:288] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:34.699280 30440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:34.700492 30440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [term 1 LEADER]: Becoming Leader. State: Replica: 8b5376da67144133907d832e0847782f, State: Running, Role: LEADER
I20260928 10:49:34.701030 30440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5376da67144133907d832e0847782f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45113 } }
I20260928 10:49:34.701431 30430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:34.703890 30441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b5376da67144133907d832e0847782f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5376da67144133907d832e0847782f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45113 } } }
I20260928 10:49:34.703864 30442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b5376da67144133907d832e0847782f. Latest consensus state: current_term: 1 leader_uuid: "8b5376da67144133907d832e0847782f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5376da67144133907d832e0847782f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45113 } } }
I20260928 10:49:34.704341 30441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:34.704341 30442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:34.707417 30447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:34.713325 30447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:34.723929 30447 catalog_manager.cc:1347] Generated new cluster ID: 732adc3b6cb041729e7fc594a9706b8d
I20260928 10:49:34.724078 30447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:34.750399 30447 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:34.750892 30447 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:34.764094 30447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b5376da67144133907d832e0847782f: Generated new TSK 0
I20260928 10:49:34.765002 30447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:34.772099 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30363
W20260928 10:49:34.773466 30425 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.39:34473
W20260928 10:49:34.774174 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:34.774328 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30363
2026-09-28T10:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:34 dist-test-slave-mn38 krb5kdc[30461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:34 dist-test-slave-mn38 krb5kdc[30461](info): set up 2 sockets
Sep 28 10:49:34 dist-test-slave-mn38 krb5kdc[30461](info): commencing operation
krb5kdc: starting...
W20260928 10:49:35.806615 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:35 dist-test-slave-mn38 krb5kdc[30461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:36.073484 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:36.419668 30474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:36.420038 30474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:36.420176 30474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:36.427686 30474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:36.427834 30474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:36.427917 30474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:36.428000 30474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:36.428064 30474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:36.438304 30474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:36.439742 30474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:36.441104 30474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:36.457864 30482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:36.457821 30479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:36.457886 30480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:36.458935 30474 server_base.cc:1034] running on GCE node
I20260928 10:49:36.459846 30474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:36.461526 30474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:36.462779 30474 hybrid_clock.cc:648] HybridClock initialized: now 1790592576462741 us; error 38 us; skew 500 ppm
I20260928 10:49:36.463177 30474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:36 dist-test-slave-mn38 krb5kdc[30461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:36.466446 30474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:36.468283 30474 webserver.cc:460] Webserver started at http://127.0.0.1:44995/ using document root <none> and password file <none>
I20260928 10:49:36.469095 30474 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:36.469242 30474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:36.469624 30474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:36.473302 30474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "7e3820acde9845eb908981ab9b05a47d"
format_stamp: "Formatted at 2026-09-28 10:49:36 on dist-test-slave-mn38"
server_key: "21ca5191ed0437ac1ba7b33cfd92716c"
server_key_iv: "4177484900fe298b9c7568e4d60c5ed3"
server_key_version: "encryptionkey@0"
I20260928 10:49:36.474318 30474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "7e3820acde9845eb908981ab9b05a47d"
format_stamp: "Formatted at 2026-09-28 10:49:36 on dist-test-slave-mn38"
server_key: "21ca5191ed0437ac1ba7b33cfd92716c"
server_key_iv: "4177484900fe298b9c7568e4d60c5ed3"
server_key_version: "encryptionkey@0"
I20260928 10:49:36.480772 30474 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 10:49:36.485134 30489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:36.486968 30474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:49:36.487195 30474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "7e3820acde9845eb908981ab9b05a47d"
format_stamp: "Formatted at 2026-09-28 10:49:36 on dist-test-slave-mn38"
server_key: "21ca5191ed0437ac1ba7b33cfd92716c"
server_key_iv: "4177484900fe298b9c7568e4d60c5ed3"
server_key_version: "encryptionkey@0"
I20260928 10:49:36.487424 30474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:36.505191 30474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:36.510087 30474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:36.510372 30474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:36.530117 30474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41171
I20260928 10:49:36.530177 30540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41171 every 8 connection(s)
I20260928 10:49:36.531705 30474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:36.535964 30541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:36.539090 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30474
I20260928 10:49:36.539280 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:36.539798 9067 external_mini_cluster.cc:1428] Setting key 0be07bbbc72e1d86318d9916d7b85b46
I20260928 10:49:36.549312 30541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d: Bootstrap starting.
I20260928 10:49:36.553845 30541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:36.555452 30541 log.cc:824] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:49:36 dist-test-slave-mn38 krb5kdc[30461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 10:49:36.558948 30541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d: No bootstrap required, opened a new log
I20260928 10:49:36.565721 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:36.541868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:33389 (local address 127.0.0.1:41171)
0928 10:49:36.542640 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:36.542653 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:36.542696 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:49:36.544728 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:36.544753 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:36.545237 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:36.545669 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:36.547384 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:36.548820 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:36.550817 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:36.551701 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:36.551887 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:36.557503 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:36.557514 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:36.557531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:36.557579 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:36.560760 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:36.561757 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:36.561765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:36.561776 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:36.561987 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:36.562512 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:36.562517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:36.562519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:36.563180 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:36.563553 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:36.564101 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 10:49:36.564477 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":344,"threads_started":1}
I20260928 10:49:36.567231 30541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3820acde9845eb908981ab9b05a47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260928 10:49:36.567607 30541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:36.567705 30541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3820acde9845eb908981ab9b05a47d, State: Initialized, Role: FOLLOWER
I20260928 10:49:36.568310 30541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3820acde9845eb908981ab9b05a47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260928 10:49:36.568563 30541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:36.568684 30541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:36.568888 30541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:36.569770 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:36.571331 30541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3820acde9845eb908981ab9b05a47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260928 10:49:36.571866 30541 leader_election.cc:302] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e3820acde9845eb908981ab9b05a47d; no voters:
I20260928 10:49:36.572468 30541 leader_election.cc:288] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:36.572606 30548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:36.573210 30548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [term 1 LEADER]: Becoming Leader. State: Replica: 7e3820acde9845eb908981ab9b05a47d, State: Running, Role: LEADER
I20260928 10:49:36.573930 30548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3820acde9845eb908981ab9b05a47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260928 10:49:36.576078 30548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e3820acde9845eb908981ab9b05a47d. Latest consensus state: current_term: 1 leader_uuid: "7e3820acde9845eb908981ab9b05a47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3820acde9845eb908981ab9b05a47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260928 10:49:36.577714 30548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:36.577322 30551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e3820acde9845eb908981ab9b05a47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3820acde9845eb908981ab9b05a47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260928 10:49:36.578236 30551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:36.583675 30541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:36.583870 30553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:36.602658 30553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:36.604488 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:36.583471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:42371 (local address 127.0.0.1:41171)
0928 10:49:36.583698 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:36.583705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:36.586355 (+ 2650us) server_negotiation.cc:407] Connection header received
0928 10:49:36.586568 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:36.586574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:36.586660 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:36.586748 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:36.588550 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:36.589535 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:36.591327 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:36.591968 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:36.592105 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:36.595449 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:36.595456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:36.595459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:36.595505 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:36.599205 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:36.600632 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:36.600638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:36.600642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:36.600854 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:36.601634 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:36.601640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:36.601644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:36.602161 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:36.603574 (+ 1413us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:36.603794 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 10:49:36.604072 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 10:49:36.614095 30553 catalog_manager.cc:1347] Generated new cluster ID: a1b2457dd5f448e4b3199105471d5734
I20260928 10:49:36.614228 30553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:36.633291 30553 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:36.635284 30553 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:36.656616 30553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e3820acde9845eb908981ab9b05a47d: Generated new TSK 0
I20260928 10:49:36.657866 30553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:36.669747 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30474
I20260928 10:49:36.671073 30536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:36.766209 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30474
W20260928 10:49:36.781286 30459 connection.cc:535] client connection to 127.0.0.1:41171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41171): Connection reset by peer (error 104)
2026-09-28T10:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:36 dist-test-slave-mn38 krb5kdc[30573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:36 dist-test-slave-mn38 krb5kdc[30573](info): set up 2 sockets
Sep 28 10:49:36 dist-test-slave-mn38 krb5kdc[30573](info): commencing operation
krb5kdc: starting...
W20260928 10:49:37.814774 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:37 dist-test-slave-mn38 krb5kdc[30573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:38.073066 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:38.421638 30586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:38.421967 30586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:38.422080 30586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:38.429776 30586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:38.429915 30586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:38.429962 30586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:38.430022 30586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:38.430065 30586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:38.440485 30586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:38.441840 30586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:38.443143 30586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:38.455370 30591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:38.455307 30594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:38.455274 30592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:38.456162 30586 server_base.cc:1034] running on GCE node
I20260928 10:49:38.457005 30586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:38.458674 30586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:38.459918 30586 hybrid_clock.cc:648] HybridClock initialized: now 1790592578459856 us; error 51 us; skew 500 ppm
I20260928 10:49:38.460289 30586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:38 dist-test-slave-mn38 krb5kdc[30573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:38.463644 30586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:38.465397 30586 webserver.cc:460] Webserver started at http://127.0.0.1:33403/ using document root <none> and password file <none>
I20260928 10:49:38.466161 30586 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:38.466300 30586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:38.466607 30586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:38.469982 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "d73cc496735d450da22b15ed7a6162f2"
format_stamp: "Formatted at 2026-09-28 10:49:38 on dist-test-slave-mn38"
server_key: "ab2e318ef4040b8a3b088e20693c9048"
server_key_iv: "1774f7bbd5edf4fedd0939a91a1cba4f"
server_key_version: "encryptionkey@0"
I20260928 10:49:38.470842 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "d73cc496735d450da22b15ed7a6162f2"
format_stamp: "Formatted at 2026-09-28 10:49:38 on dist-test-slave-mn38"
server_key: "ab2e318ef4040b8a3b088e20693c9048"
server_key_iv: "1774f7bbd5edf4fedd0939a91a1cba4f"
server_key_version: "encryptionkey@0"
I20260928 10:49:38.476481 30586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 10:49:38.480150 30601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:38.481730 30586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:49:38.481910 30586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "d73cc496735d450da22b15ed7a6162f2"
format_stamp: "Formatted at 2026-09-28 10:49:38 on dist-test-slave-mn38"
server_key: "ab2e318ef4040b8a3b088e20693c9048"
server_key_iv: "1774f7bbd5edf4fedd0939a91a1cba4f"
server_key_version: "encryptionkey@0"
I20260928 10:49:38.482105 30586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:38.500795 30586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:38.506117 30586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:38.506430 30586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:38.526434 30586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37929
I20260928 10:49:38.526489 30652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37929 every 8 connection(s)
I20260928 10:49:38.527894 30586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:38.529414 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30586
I20260928 10:49:38.529626 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:38.530076 9067 external_mini_cluster.cc:1428] Setting key 81041ba4de2e21a01122a40a4316ba62
I20260928 10:49:38.532660 30653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:38.546928 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:38.532094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:33875 (local address 127.0.0.1:37929)
0928 10:49:38.532934 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:38.532952 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:38.533009 (+ 57us) server_negotiation.cc:407] Connection header received
0928 10:49:38.535354 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:38.535387 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:38.535971 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:38.536478 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:38.537054 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:38.539149 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:38.540785 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:38.541948 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:38.542146 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:38.543626 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:38.543638 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:38.543666 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:38.544489 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:38.544612 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:38.544654 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:38.544833 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:38.545180 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 10:49:38.545405 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":336,"threads_started":1}
I20260928 10:49:38.548053 30653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2: Bootstrap starting.
I20260928 10:49:38.551250 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:38.551645 30653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:38.552991 30653 log.cc:824] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:38.556954 30653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2: No bootstrap required, opened a new log
I20260928 10:49:38.563519 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:38.556584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:53155 (local address 127.0.0.1:37929)
0928 10:49:38.556828 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:38.556834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:38.557195 (+ 361us) server_negotiation.cc:407] Connection header received
0928 10:49:38.557517 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:38.557525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:38.557604 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:38.557698 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:38.558314 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:38.559103 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:38.560959 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:38.561423 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:38.561553 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:38.562749 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:38.562755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:38.562758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:38.562788 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:38.562827 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:38.562842 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:38.562923 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:38.563115 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 10:49:38.563162 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:49:38.566448 30653 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73cc496735d450da22b15ed7a6162f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } }
I20260928 10:49:38.566864 30653 raft_consensus.cc:374] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:38.566963 30653 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73cc496735d450da22b15ed7a6162f2, State: Initialized, Role: FOLLOWER
I20260928 10:49:38.567512 30653 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73cc496735d450da22b15ed7a6162f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } }
I20260928 10:49:38.567744 30653 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:38.567873 30653 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:38.568029 30653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:38.570340 30653 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73cc496735d450da22b15ed7a6162f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } }
I20260928 10:49:38.570842 30653 leader_election.cc:302] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73cc496735d450da22b15ed7a6162f2; no voters:
I20260928 10:49:38.571404 30653 leader_election.cc:288] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:38.571527 30663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:38.573102 30663 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [term 1 LEADER]: Becoming Leader. State: Replica: d73cc496735d450da22b15ed7a6162f2, State: Running, Role: LEADER
I20260928 10:49:38.573400 30653 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:38.573684 30663 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73cc496735d450da22b15ed7a6162f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } }
I20260928 10:49:38.576212 30665 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73cc496735d450da22b15ed7a6162f2. Latest consensus state: current_term: 1 leader_uuid: "d73cc496735d450da22b15ed7a6162f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73cc496735d450da22b15ed7a6162f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } } }
I20260928 10:49:38.576520 30665 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:38.576675 30664 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73cc496735d450da22b15ed7a6162f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73cc496735d450da22b15ed7a6162f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } } }
I20260928 10:49:38.576972 30664 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:38.578552 30672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:38.583056 30672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:38.593312 30672 catalog_manager.cc:1347] Generated new cluster ID: 640fe9c81d394367ba8b644ea3b62a39
I20260928 10:49:38.593492 30672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:38.610929 30672 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:38.612422 30672 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:38.629907 30672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d73cc496735d450da22b15ed7a6162f2: Generated new TSK 0
I20260928 10:49:38.630923 30672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:38.637624 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30586
W20260928 10:49:38.638883 30648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.39:33875
W20260928 10:49:38.639633 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:38.639817 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30586
W20260928 10:49:38.654902 30571 connection.cc:535] client connection to 127.0.0.1:37929 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37929): Connection reset by peer (error 104)
2026-09-28T10:49:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:38 dist-test-slave-mn38 krb5kdc[30684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:38 dist-test-slave-mn38 krb5kdc[30684](info): set up 2 sockets
Sep 28 10:49:38 dist-test-slave-mn38 krb5kdc[30684](info): commencing operation
krb5kdc: starting...
W20260928 10:49:39.670769 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:39 dist-test-slave-mn38 krb5kdc[30684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 10:49:39.928627 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:40.277716 30697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:40.278048 30697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:40.278178 30697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:40.285807 30697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:40.285928 30697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:40.285996 30697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:40.286070 30697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:40.286123 30697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:40.296648 30697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:40.298283 30697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:40.300051 30697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:40.313059 30702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:40.313196 30703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:40.313503 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:40.314448 30697 server_base.cc:1034] running on GCE node
I20260928 10:49:40.315303 30697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:40.316893 30697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:40.318131 30697 hybrid_clock.cc:648] HybridClock initialized: now 1790592580318069 us; error 49 us; skew 500 ppm
I20260928 10:49:40.318538 30697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:40 dist-test-slave-mn38 krb5kdc[30684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:40.326818 30697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 10:49:40.328702 30697 webserver.cc:460] Webserver started at http://127.0.0.1:46165/ using document root <none> and password file <none>
I20260928 10:49:40.329573 30697 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:40.329720 30697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:40.330067 30697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:40.333710 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "2c377b30feb34dfd854144d97b04dddb"
format_stamp: "Formatted at 2026-09-28 10:49:40 on dist-test-slave-mn38"
server_key: "48438db281ced67c47c549a5d12a721d"
server_key_iv: "0dd80c016f2dea8fa6a9668960cfa6e5"
server_key_version: "encryptionkey@0"
I20260928 10:49:40.334877 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "2c377b30feb34dfd854144d97b04dddb"
format_stamp: "Formatted at 2026-09-28 10:49:40 on dist-test-slave-mn38"
server_key: "48438db281ced67c47c549a5d12a721d"
server_key_iv: "0dd80c016f2dea8fa6a9668960cfa6e5"
server_key_version: "encryptionkey@0"
I20260928 10:49:40.342965 30697 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260928 10:49:40.346863 30712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:40.348593 30697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:40.348824 30697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "2c377b30feb34dfd854144d97b04dddb"
format_stamp: "Formatted at 2026-09-28 10:49:40 on dist-test-slave-mn38"
server_key: "48438db281ced67c47c549a5d12a721d"
server_key_iv: "0dd80c016f2dea8fa6a9668960cfa6e5"
server_key_version: "encryptionkey@0"
I20260928 10:49:40.349045 30697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:40.355361 30697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:40.359859 30697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:40.360142 30697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:40.378147 30697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46867
I20260928 10:49:40.378185 30763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46867 every 8 connection(s)
I20260928 10:49:40.379561 30697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:40.383373 30764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:40.384045 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30697
I20260928 10:49:40.384207 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:40.384661 9067 external_mini_cluster.cc:1428] Setting key 6269a798abe4fc566def638ffb005837
I20260928 10:49:40.394701 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:40.386594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:58575 (local address 127.0.0.1:46867)
0928 10:49:40.387535 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:40.387553 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:40.387611 (+ 58us) server_negotiation.cc:407] Connection header received
0928 10:49:40.390525 (+ 2914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:40.390565 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:40.391154 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:40.391707 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:40.392698 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:40.392707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:40.392724 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:40.392742 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:40.392813 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:40.392845 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:40.393061 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:40.393194 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 10:49:40.393409 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":383,"threads_started":1}
I20260928 10:49:40.398407 30764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb: Bootstrap starting.
I20260928 10:49:40.398576 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:40.402964 30764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:40.404799 30764 log.cc:824] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:40.407775 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:40.404490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:46989 (local address 127.0.0.1:46867)
0928 10:49:40.404721 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:40.404728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:40.404979 (+ 251us) server_negotiation.cc:407] Connection header received
0928 10:49:40.405188 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:40.405195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:40.405273 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:40.405708 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:40.406923 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:40.406930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:40.406935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:40.406959 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:40.407014 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:40.407041 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:40.407189 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:40.407398 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 10:49:40.407473 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:49:40.409538 30764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb: No bootstrap required, opened a new log
I20260928 10:49:40.418872 30764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c377b30feb34dfd854144d97b04dddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46867 } }
I20260928 10:49:40.419394 30764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:40.419512 30764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c377b30feb34dfd854144d97b04dddb, State: Initialized, Role: FOLLOWER
I20260928 10:49:40.420228 30764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c377b30feb34dfd854144d97b04dddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46867 } }
I20260928 10:49:40.420521 30764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:40.420672 30764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:40.420855 30764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:40.423539 30764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c377b30feb34dfd854144d97b04dddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46867 } }
I20260928 10:49:40.424203 30764 leader_election.cc:302] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c377b30feb34dfd854144d97b04dddb; no voters:
I20260928 10:49:40.424955 30764 leader_election.cc:288] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:40.425246 30774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:40.426429 30774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [term 1 LEADER]: Becoming Leader. State: Replica: 2c377b30feb34dfd854144d97b04dddb, State: Running, Role: LEADER
I20260928 10:49:40.427050 30774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c377b30feb34dfd854144d97b04dddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46867 } }
I20260928 10:49:40.427525 30764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:40.429956 30775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c377b30feb34dfd854144d97b04dddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c377b30feb34dfd854144d97b04dddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46867 } } }
I20260928 10:49:40.430267 30776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c377b30feb34dfd854144d97b04dddb. Latest consensus state: current_term: 1 leader_uuid: "2c377b30feb34dfd854144d97b04dddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c377b30feb34dfd854144d97b04dddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46867 } } }
I20260928 10:49:40.430469 30775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:40.430538 30776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:40.431564 30783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:40.437112 30783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:40.449226 30783 catalog_manager.cc:1347] Generated new cluster ID: e43245d3eabc45bf8e9d2ebd8e22f9d9
I20260928 10:49:40.449364 30783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:40.487277 30783 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:40.487836 30783 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:40.504093 30783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c377b30feb34dfd854144d97b04dddb: Generated new TSK 0
I20260928 10:49:40.505023 30783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:40.520457 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30697
W20260928 10:49:40.522426 30759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.39:58575
W20260928 10:49:40.523245 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:40.523411 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30697
2026-09-28T10:49:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:40 dist-test-slave-mn38 krb5kdc[30795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:40 dist-test-slave-mn38 krb5kdc[30795](info): set up 2 sockets
Sep 28 10:49:40 dist-test-slave-mn38 krb5kdc[30795](info): commencing operation
krb5kdc: starting...
W20260928 10:49:41.558743 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:41 dist-test-slave-mn38 krb5kdc[30795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:41.821493 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36053
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:42.173858 30808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:42.174172 30808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:42.174307 30808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:42.182849 30808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:42.183007 30808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:42.183079 30808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:42.183173 30808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:42.183223 30808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:42.195317 30808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:36053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:42.197077 30808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:42.198649 30808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:42.214169 30816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:42.214257 30814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:42.214357 30813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:42.214959 30808 server_base.cc:1034] running on GCE node
I20260928 10:49:42.216027 30808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:42.217840 30808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:42.219152 30808 hybrid_clock.cc:648] HybridClock initialized: now 1790592582219110 us; error 41 us; skew 500 ppm
I20260928 10:49:42.219641 30808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:42 dist-test-slave-mn38 krb5kdc[30795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592582, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:42.223783 30808 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:42.225911 30808 webserver.cc:460] Webserver started at http://127.8.218.254:45025/ using document root <none> and password file <none>
I20260928 10:49:42.227035 30808 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:42.227300 30808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:42.227774 30808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:42.232004 30808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "33bafea4524c486b83db4e663b7dece4"
format_stamp: "Formatted at 2026-09-28 10:49:42 on dist-test-slave-mn38"
server_key: "b8d9fd428c1856652da4352d8cbd0d00"
server_key_iv: "158e804d51e7d303515157df8bec7a7a"
server_key_version: "encryptionkey@0"
I20260928 10:49:42.233096 30808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "33bafea4524c486b83db4e663b7dece4"
format_stamp: "Formatted at 2026-09-28 10:49:42 on dist-test-slave-mn38"
server_key: "b8d9fd428c1856652da4352d8cbd0d00"
server_key_iv: "158e804d51e7d303515157df8bec7a7a"
server_key_version: "encryptionkey@0"
I20260928 10:49:42.240605 30808 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:49:42.245026 30823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:42.246762 30808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:49:42.247014 30808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "33bafea4524c486b83db4e663b7dece4"
format_stamp: "Formatted at 2026-09-28 10:49:42 on dist-test-slave-mn38"
server_key: "b8d9fd428c1856652da4352d8cbd0d00"
server_key_iv: "158e804d51e7d303515157df8bec7a7a"
server_key_version: "encryptionkey@0"
I20260928 10:49:42.247241 30808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:42.274319 30808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:42.278790 30808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:42.279062 30808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:42.298305 30808 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:36053
I20260928 10:49:42.298316 30874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:36053 every 8 connection(s)
I20260928 10:49:42.299724 30808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:42.303889 30875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:42.308298 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30808
I20260928 10:49:42.308485 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:42.309048 9067 external_mini_cluster.cc:1428] Setting key 92f3d768a6327c4f078e1f07a697272a
I20260928 10:49:42.317430 30875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4: Bootstrap starting.
I20260928 10:49:42.321501 30875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:42.322676 30875 log.cc:824] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:49:42 dist-test-slave-mn38 krb5kdc[30795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:42.326478 30875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4: No bootstrap required, opened a new log
I20260928 10:49:42.335152 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:42.310976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46783 (local address 127.8.218.254:36053)
0928 10:49:42.311794 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:42.311811 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:42.311854 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:49:42.313607 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:42.313635 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:42.314135 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:42.314569 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:42.316059 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:42.317527 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:42.318955 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:42.320102 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:42.320325 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:42.324897 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:42.324907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:42.324926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:42.324973 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:42.328962 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:42.330260 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:42.330277 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:42.330289 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:42.330588 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:42.331216 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:42.331221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:42.331224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:42.332081 (+ 857us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:42.332459 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:42.333019 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 10:49:42.333537 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":351,"threads_started":1}
I20260928 10:49:42.337683 30875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bafea4524c486b83db4e663b7dece4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36053 } }
I20260928 10:49:42.338131 30875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:42.338271 30875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33bafea4524c486b83db4e663b7dece4, State: Initialized, Role: FOLLOWER
I20260928 10:49:42.338989 30875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bafea4524c486b83db4e663b7dece4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36053 } }
I20260928 10:49:42.339285 30875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:42.339423 30875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:42.339636 30875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:42.340615 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:42.342300 30875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bafea4524c486b83db4e663b7dece4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36053 } }
I20260928 10:49:42.342909 30875 leader_election.cc:302] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33bafea4524c486b83db4e663b7dece4; no voters:
I20260928 10:49:42.343657 30875 leader_election.cc:288] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:42.343804 30882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:42.346011 30875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:42.350768 30882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [term 1 LEADER]: Becoming Leader. State: Replica: 33bafea4524c486b83db4e663b7dece4, State: Running, Role: LEADER
I20260928 10:49:42.351500 30882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bafea4524c486b83db4e663b7dece4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36053 } }
I20260928 10:49:42.354820 30886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33bafea4524c486b83db4e663b7dece4. Latest consensus state: current_term: 1 leader_uuid: "33bafea4524c486b83db4e663b7dece4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bafea4524c486b83db4e663b7dece4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36053 } } }
I20260928 10:49:42.355315 30886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:42.355777 30885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33bafea4524c486b83db4e663b7dece4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bafea4524c486b83db4e663b7dece4" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 36053 } } }
I20260928 10:49:42.356098 30885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4 [sys.catalog]: This master's current role is: LEADER
W20260928 10:49:42.367192 30899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:49:42.367313 30899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:49:42.367440 30900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:42.369530 30900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:42.370067 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:42.351670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38957 (local address 127.8.218.254:36053)
0928 10:49:42.351863 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:42.351871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:42.352483 (+ 612us) server_negotiation.cc:407] Connection header received
0928 10:49:42.352694 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:42.352699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:42.352763 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:42.352852 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:42.354943 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:42.356124 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:42.358386 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:42.359190 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:42.359307 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:42.362977 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:42.362984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:42.362988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:42.363026 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:42.366675 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:42.367629 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:42.367635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:42.367638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:42.367803 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:42.368351 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:42.368357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:42.368360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:42.368806 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:42.369065 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:42.369526 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 10:49:42.369716 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:49:42.377976 30900 catalog_manager.cc:1347] Generated new cluster ID: c17ec1da500b4eb7a6fab54dea048381
I20260928 10:49:42.378098 30900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:42.397869 30900 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:42.399345 30900 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:42.407526 30900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33bafea4524c486b83db4e663b7dece4: Generated new TSK 0
I20260928 10:49:42.408870 30900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:42.417037 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30808
I20260928 10:49:42.418475 30870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:42.529300 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30808
W20260928 10:49:42.544241 30793 connection.cc:535] client connection to 127.8.218.254:36053 recv error: Network error: failed to read from TLS socket (remote: 127.8.218.254:36053): Connection reset by peer (error 104)
2026-09-28T10:49:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:42 dist-test-slave-mn38 krb5kdc[30907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:42 dist-test-slave-mn38 krb5kdc[30907](info): set up 2 sockets
Sep 28 10:49:42 dist-test-slave-mn38 krb5kdc[30907](info): commencing operation
krb5kdc: starting...
W20260928 10:49:43.550596 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:43 dist-test-slave-mn38 krb5kdc[30907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:43.814100 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:41769
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:41769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:44.159970 30920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:44.160372 30920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:44.160531 30920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:44.168785 30920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:44.168953 30920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:44.169044 30920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:44.169152 30920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:44.169214 30920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:44.179760 30920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:41769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:41769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:44.181198 30920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:44.182623 30920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:44.195595 30925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:44.195564 30926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:44.195634 30928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:44.196120 30920 server_base.cc:1034] running on GCE node
I20260928 10:49:44.197233 30920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:44.198834 30920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:44.200083 30920 hybrid_clock.cc:648] HybridClock initialized: now 1790592584200018 us; error 60 us; skew 500 ppm
I20260928 10:49:44.200498 30920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:44 dist-test-slave-mn38 krb5kdc[30907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592584, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:44.203989 30920 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:44.205915 30920 webserver.cc:460] Webserver started at http://127.8.218.254:33299/ using document root <none> and password file <none>
I20260928 10:49:44.206698 30920 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:44.206838 30920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:44.207173 30920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:44.210570 30920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "1caff29135d54d2bb46a45d436998661"
format_stamp: "Formatted at 2026-09-28 10:49:44 on dist-test-slave-mn38"
server_key: "b2b39af3bb805cf2bd21e50df4f7b444"
server_key_iv: "526e4110b119caeec29497d5e28e0dcf"
server_key_version: "encryptionkey@0"
I20260928 10:49:44.211440 30920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "1caff29135d54d2bb46a45d436998661"
format_stamp: "Formatted at 2026-09-28 10:49:44 on dist-test-slave-mn38"
server_key: "b2b39af3bb805cf2bd21e50df4f7b444"
server_key_iv: "526e4110b119caeec29497d5e28e0dcf"
server_key_version: "encryptionkey@0"
I20260928 10:49:44.217286 30920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 10:49:44.221544 30935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:44.223367 30920 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:49:44.223590 30920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "1caff29135d54d2bb46a45d436998661"
format_stamp: "Formatted at 2026-09-28 10:49:44 on dist-test-slave-mn38"
server_key: "b2b39af3bb805cf2bd21e50df4f7b444"
server_key_iv: "526e4110b119caeec29497d5e28e0dcf"
server_key_version: "encryptionkey@0"
I20260928 10:49:44.223812 30920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:44.254268 30920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:44.260090 30920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:44.260426 30920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:44.288007 30920 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:41769
I20260928 10:49:44.288007 30986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:41769 every 8 connection(s)
I20260928 10:49:44.289867 30920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:44.291157 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 30920
I20260928 10:49:44.291337 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:44.291749 9067 external_mini_cluster.cc:1428] Setting key 9899b0d991aa76d8970bcf27dedd9e6e
I20260928 10:49:44.295157 30987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:44.306569 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:44.293825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44229 (local address 127.8.218.254:41769)
0928 10:49:44.294754 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:44.294768 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:44.294825 (+ 57us) server_negotiation.cc:407] Connection header received
0928 10:49:44.296546 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:44.296572 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:44.297041 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:44.297677 (+ 636us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:44.298249 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:44.299699 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:44.301682 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:44.302463 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:44.302607 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:44.303973 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:44.303981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:44.303998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:44.304610 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:44.304693 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:44.304724 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:44.304886 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:44.305170 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 10:49:44.305490 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":383,"threads_started":1}
I20260928 10:49:44.308214 30987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661: Bootstrap starting.
I20260928 10:49:44.310915 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:44.311697 30987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:44.312757 30987 log.cc:824] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:44.316368 30987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661: No bootstrap required, opened a new log
I20260928 10:49:44.321375 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:44.315326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48395 (local address 127.8.218.254:41769)
0928 10:49:44.315475 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:44.315480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:44.315957 (+ 477us) server_negotiation.cc:407] Connection header received
0928 10:49:44.316145 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:44.316149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:44.316213 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:44.316298 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:44.316850 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:44.317666 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:44.318972 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:44.319454 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:44.319564 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:44.320675 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:44.320680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:44.320683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:44.320702 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:44.320740 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:44.320759 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:44.320857 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:44.321059 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 10:49:44.321118 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:49:44.324482 30987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caff29135d54d2bb46a45d436998661" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 41769 } }
I20260928 10:49:44.324874 30987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:44.324975 30987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1caff29135d54d2bb46a45d436998661, State: Initialized, Role: FOLLOWER
I20260928 10:49:44.325656 30987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caff29135d54d2bb46a45d436998661" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 41769 } }
I20260928 10:49:44.325949 30987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:44.326098 30987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:44.326289 30987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:44.328706 30987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caff29135d54d2bb46a45d436998661" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 41769 } }
I20260928 10:49:44.329255 30987 leader_election.cc:302] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1caff29135d54d2bb46a45d436998661; no voters:
I20260928 10:49:44.329874 30987 leader_election.cc:288] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:44.330013 30997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:44.331288 30997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [term 1 LEADER]: Becoming Leader. State: Replica: 1caff29135d54d2bb46a45d436998661, State: Running, Role: LEADER
I20260928 10:49:44.331890 30997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caff29135d54d2bb46a45d436998661" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 41769 } }
I20260928 10:49:44.332369 30987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:44.335282 30998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1caff29135d54d2bb46a45d436998661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caff29135d54d2bb46a45d436998661" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 41769 } } }
I20260928 10:49:44.335269 30999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1caff29135d54d2bb46a45d436998661. Latest consensus state: current_term: 1 leader_uuid: "1caff29135d54d2bb46a45d436998661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caff29135d54d2bb46a45d436998661" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 41769 } } }
I20260928 10:49:44.335813 30999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:44.335814 30998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:44.337707 31003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:44.343484 31003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:44.353605 31003 catalog_manager.cc:1347] Generated new cluster ID: f05f248997e944fab0c4b4923d1b2aca
I20260928 10:49:44.353727 31003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:44.364859 31003 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:44.366367 31003 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:44.379235 31003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1caff29135d54d2bb46a45d436998661: Generated new TSK 0
I20260928 10:49:44.380278 31003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:44.388645 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30920
W20260928 10:49:44.390401 30982 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44229
W20260928 10:49:44.391211 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 30920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:44.391362 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 30920
W20260928 10:49:44.405932 30905 connection.cc:535] client connection to 127.8.218.254:41769 recv error: Network error: failed to read from TLS socket (remote: 127.8.218.254:41769): Connection reset by peer (error 104)
2026-09-28T10:49:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1861 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:44 dist-test-slave-mn38 krb5kdc[31018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:44 dist-test-slave-mn38 krb5kdc[31018](info): set up 2 sockets
Sep 28 10:49:44 dist-test-slave-mn38 krb5kdc[31018](info): commencing operation
krb5kdc: starting...
W20260928 10:49:45.422698 9067 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:45 dist-test-slave-mn38 krb5kdc[31018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:45.687438 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:33557
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:46.033826 31031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:46.034137 31031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:46.034273 31031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:46.041810 31031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:46.041935 31031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:46.041993 31031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:46.042074 31031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:46.042135 31031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:46.053428 31031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:33557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:46.054894 31031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:46.056341 31031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:46.068426 31036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:46.068450 31037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:46.068966 31039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:46.069310 31031 server_base.cc:1034] running on GCE node
I20260928 10:49:46.070098 31031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:46.071564 31031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:46.072801 31031 hybrid_clock.cc:648] HybridClock initialized: now 1790592586072734 us; error 56 us; skew 500 ppm
I20260928 10:49:46.073248 31031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:46 dist-test-slave-mn38 krb5kdc[31018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:46.082953 31031 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:46.084607 31031 webserver.cc:460] Webserver started at http://127.8.218.254:34977/ using document root <none> and password file <none>
I20260928 10:49:46.085337 31031 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:46.085493 31031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:46.085796 31031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:46.089008 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "cb5705c19e7b4c3cb6acc09da2dd198d"
format_stamp: "Formatted at 2026-09-28 10:49:46 on dist-test-slave-mn38"
server_key: "49ade1b28de1646d66f268a221480a4d"
server_key_iv: "728ea43945fc28f24b38aa072d9651a8"
server_key_version: "encryptionkey@0"
I20260928 10:49:46.089908 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "cb5705c19e7b4c3cb6acc09da2dd198d"
format_stamp: "Formatted at 2026-09-28 10:49:46 on dist-test-slave-mn38"
server_key: "49ade1b28de1646d66f268a221480a4d"
server_key_iv: "728ea43945fc28f24b38aa072d9651a8"
server_key_version: "encryptionkey@0"
I20260928 10:49:46.095743 31031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:49:46.099707 31046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:46.101233 31031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 10:49:46.101464 31031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "cb5705c19e7b4c3cb6acc09da2dd198d"
format_stamp: "Formatted at 2026-09-28 10:49:46 on dist-test-slave-mn38"
server_key: "49ade1b28de1646d66f268a221480a4d"
server_key_iv: "728ea43945fc28f24b38aa072d9651a8"
server_key_version: "encryptionkey@0"
I20260928 10:49:46.101703 31031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:46.108398 31031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:46.113354 31031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:46.113708 31031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:46.132799 31031 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:33557
I20260928 10:49:46.132845 31097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:33557 every 8 connection(s)
I20260928 10:49:46.134395 31031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:46.138667 31098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:46.143592 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31031
I20260928 10:49:46.143777 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:46.144196 9067 external_mini_cluster.cc:1428] Setting key 6387cb98a7cb4e474cd842880b622067
I20260928 10:49:46.150998 31098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d: Bootstrap starting.
I20260928 10:49:46.153053 31101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:46.146259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37029 (local address 127.8.218.254:33557)
0928 10:49:46.147080 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:46.147094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:46.147137 (+ 43us) server_negotiation.cc:407] Connection header received
0928 10:49:46.149201 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:46.149230 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:46.149687 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:46.150066 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:46.151342 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:46.151350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:46.151365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:46.151383 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:46.151458 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:46.151480 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:46.151674 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:46.151800 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 10:49:46.152030 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":354,"threads_started":1}
I20260928 10:49:46.154628 31098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:46.155776 31098 log.cc:824] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:46.157004 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:46.159304 31098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d: No bootstrap required, opened a new log
I20260928 10:49:46.164781 31101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:46.162320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50479 (local address 127.8.218.254:33557)
0928 10:49:46.162523 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:46.162530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:46.162909 (+ 379us) server_negotiation.cc:407] Connection header received
0928 10:49:46.163063 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:46.163069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:46.163147 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:46.163275 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:46.164114 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:46.164120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:46.164125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:46.164144 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:46.164198 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:46.164217 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:46.164342 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:46.164390 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 10:49:46.164450 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:49:46.168301 31098 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5705c19e7b4c3cb6acc09da2dd198d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33557 } }
I20260928 10:49:46.168699 31098 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:46.168807 31098 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5705c19e7b4c3cb6acc09da2dd198d, State: Initialized, Role: FOLLOWER
I20260928 10:49:46.169409 31098 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5705c19e7b4c3cb6acc09da2dd198d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33557 } }
I20260928 10:49:46.169705 31098 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:46.169845 31098 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:46.170013 31098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:46.172308 31098 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5705c19e7b4c3cb6acc09da2dd198d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33557 } }
I20260928 10:49:46.172837 31098 leader_election.cc:302] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5705c19e7b4c3cb6acc09da2dd198d; no voters:
I20260928 10:49:46.173403 31098 leader_election.cc:288] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:46.173560 31108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:46.174178 31108 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [term 1 LEADER]: Becoming Leader. State: Replica: cb5705c19e7b4c3cb6acc09da2dd198d, State: Running, Role: LEADER
I20260928 10:49:46.174794 31108 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5705c19e7b4c3cb6acc09da2dd198d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33557 } }
I20260928 10:49:46.175691 31098 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:46.177790 31110 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5705c19e7b4c3cb6acc09da2dd198d. Latest consensus state: current_term: 1 leader_uuid: "cb5705c19e7b4c3cb6acc09da2dd198d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5705c19e7b4c3cb6acc09da2dd198d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33557 } } }
I20260928 10:49:46.178218 31110 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:46.178925 31109 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5705c19e7b4c3cb6acc09da2dd198d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5705c19e7b4c3cb6acc09da2dd198d" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 33557 } } }
I20260928 10:49:46.179237 31109 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:46.183281 31115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:46.188129 31115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:46.198904 31115 catalog_manager.cc:1347] Generated new cluster ID: c3d1b1337eb04205a6602d98cde1c181
I20260928 10:49:46.199059 31115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:46.216351 31115 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:46.216872 31115 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:46.226013 31115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5705c19e7b4c3cb6acc09da2dd198d: Generated new TSK 0
I20260928 10:49:46.226984 31115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:46.236828 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31031
W20260928 10:49:46.238327 31093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37029
W20260928 10:49:46.239192 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31031: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:46.239365 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31031
2026-09-28T10:49:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:46 dist-test-slave-mn38 krb5kdc[31129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:46 dist-test-slave-mn38 krb5kdc[31129](info): set up 2 sockets
Sep 28 10:49:46 dist-test-slave-mn38 krb5kdc[31129](info): commencing operation
krb5kdc: starting...
W20260928 10:49:47.278640 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:47 dist-test-slave-mn38 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:47.541277 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37159
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:37159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:47.885596 31142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:47.885919 31142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:47.886047 31142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:47.893819 31142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:47.893961 31142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:47.894034 31142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:47.894121 31142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:47.894179 31142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:47.904662 31142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:37159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:47.906196 31142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:47.907639 31142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:47.920022 31147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:47.919989 31150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:47.919946 31148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:47.920871 31142 server_base.cc:1034] running on GCE node
I20260928 10:49:47.921756 31142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:47.923393 31142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:47.924659 31142 hybrid_clock.cc:648] HybridClock initialized: now 1790592587924604 us; error 42 us; skew 500 ppm
I20260928 10:49:47.925040 31142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:47 dist-test-slave-mn38 krb5kdc[31129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:47.928344 31142 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:47.929981 31142 webserver.cc:460] Webserver started at http://127.8.218.254:41303/ using document root <none> and password file <none>
I20260928 10:49:47.930696 31142 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:47.930826 31142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:47.931144 31142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:47.934513 31142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "e9bd09a8cbac4e0faca340159ce3ed0f"
format_stamp: "Formatted at 2026-09-28 10:49:47 on dist-test-slave-mn38"
server_key: "be8754139d8f671ec90e59e956f0bd69"
server_key_iv: "a2081669ef4be7060642c6330f716f08"
server_key_version: "encryptionkey@0"
I20260928 10:49:47.935369 31142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "e9bd09a8cbac4e0faca340159ce3ed0f"
format_stamp: "Formatted at 2026-09-28 10:49:47 on dist-test-slave-mn38"
server_key: "be8754139d8f671ec90e59e956f0bd69"
server_key_iv: "a2081669ef4be7060642c6330f716f08"
server_key_version: "encryptionkey@0"
I20260928 10:49:47.941501 31142 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:49:47.945647 31157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:47.947484 31142 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 10:49:47.947700 31142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "e9bd09a8cbac4e0faca340159ce3ed0f"
format_stamp: "Formatted at 2026-09-28 10:49:47 on dist-test-slave-mn38"
server_key: "be8754139d8f671ec90e59e956f0bd69"
server_key_iv: "a2081669ef4be7060642c6330f716f08"
server_key_version: "encryptionkey@0"
I20260928 10:49:47.947919 31142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:47.985411 31142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:47.990206 31142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:47.990525 31142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:48.010322 31142 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:37159
I20260928 10:49:48.010337 31208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:37159 every 8 connection(s)
I20260928 10:49:48.011793 31142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:48.015954 31209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:48.017711 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31142
I20260928 10:49:48.017885 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:48.018464 9067 external_mini_cluster.cc:1428] Setting key 94ad7e39b7a54d34e32473c37cda9743
I20260928 10:49:48.031205 31209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f: Bootstrap starting.
Sep 28 10:49:48 dist-test-slave-mn38 krb5kdc[31129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:48.035135 31209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:48.036279 31209 log.cc:824] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:48.039850 31209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f: No bootstrap required, opened a new log
I20260928 10:49:48.044077 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:48.020487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55469 (local address 127.8.218.254:37159)
0928 10:49:48.021342 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:48.021357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:48.021408 (+ 51us) server_negotiation.cc:407] Connection header received
0928 10:49:48.023130 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:48.023156 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:48.023643 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:48.024009 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:48.025911 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:48.027310 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:48.029138 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:48.029971 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:48.030111 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:48.035061 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:48.035070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:48.035088 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:48.035143 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:48.038827 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:48.039677 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:48.039688 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:48.039696 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:48.039936 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:48.040332 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:48.040341 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:48.040345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:48.041138 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:48.041484 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:48.041991 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 10:49:48.042514 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":390,"threads_started":1}
I20260928 10:49:48.048815 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:48.050109 31209 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd09a8cbac4e0faca340159ce3ed0f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37159 } }
I20260928 10:49:48.050629 31209 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:48.050741 31209 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bd09a8cbac4e0faca340159ce3ed0f, State: Initialized, Role: FOLLOWER
I20260928 10:49:48.051469 31209 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd09a8cbac4e0faca340159ce3ed0f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37159 } }
I20260928 10:49:48.051795 31209 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:48.051957 31209 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:48.052183 31209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:48.055433 31209 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd09a8cbac4e0faca340159ce3ed0f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37159 } }
I20260928 10:49:48.056083 31209 leader_election.cc:302] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bd09a8cbac4e0faca340159ce3ed0f; no voters:
I20260928 10:49:48.056828 31209 leader_election.cc:288] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:48.056945 31219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:48.058173 31219 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [term 1 LEADER]: Becoming Leader. State: Replica: e9bd09a8cbac4e0faca340159ce3ed0f, State: Running, Role: LEADER
I20260928 10:49:48.058871 31219 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd09a8cbac4e0faca340159ce3ed0f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37159 } }
I20260928 10:49:48.059414 31209 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:48.061944 31220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bd09a8cbac4e0faca340159ce3ed0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd09a8cbac4e0faca340159ce3ed0f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37159 } } }
I20260928 10:49:48.062361 31220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:48.066161 31221 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bd09a8cbac4e0faca340159ce3ed0f. Latest consensus state: current_term: 1 leader_uuid: "e9bd09a8cbac4e0faca340159ce3ed0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd09a8cbac4e0faca340159ce3ed0f" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 37159 } } }
I20260928 10:49:48.067305 31221 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:48.068842 31224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:48.071600 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:48.053924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53945 (local address 127.8.218.254:37159)
0928 10:49:48.054119 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:48.054125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:48.054540 (+ 415us) server_negotiation.cc:407] Connection header received
0928 10:49:48.054716 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:48.054721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:48.054794 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:48.054883 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:48.056458 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:48.057401 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:48.059016 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:48.059556 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:48.059680 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:48.062721 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:48.062728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:48.062733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:48.062778 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:48.067094 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:48.068198 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:48.068203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:48.068206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:48.068398 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:48.068932 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:48.068937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:48.068940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:48.069387 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:48.070662 (+ 1275us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:48.070874 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 10:49:48.071163 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 10:49:48.075265 31224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:48.085083 31224 catalog_manager.cc:1347] Generated new cluster ID: 7e4d10afa41447ea81a8943183140751
I20260928 10:49:48.085201 31224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:48.103497 31224 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:48.105643 31224 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:48.115335 31224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bd09a8cbac4e0faca340159ce3ed0f: Generated new TSK 0
I20260928 10:49:48.116506 31224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:48.128273 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31142
I20260928 10:49:48.129495 31204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:48.223646 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31142
2026-09-28T10:49:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:48 dist-test-slave-mn38 krb5kdc[31241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:48 dist-test-slave-mn38 krb5kdc[31241](info): set up 2 sockets
Sep 28 10:49:48 dist-test-slave-mn38 krb5kdc[31241](info): commencing operation
krb5kdc: starting...
W20260928 10:49:49.258880 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:49 dist-test-slave-mn38 krb5kdc[31241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:49.525197 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:39281
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:39281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:49.873806 31254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:49.874117 31254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:49.874212 31254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:49.882150 31254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:49.882299 31254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:49.882349 31254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:49.882411 31254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:49.882457 31254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:49.892743 31254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:39281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:39281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:49.894153 31254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:49.895503 31254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:49.908947 31259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:49.908986 31260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:49.909016 31262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:49.909669 31254 server_base.cc:1034] running on GCE node
I20260928 10:49:49.910645 31254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:49.912375 31254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:49.913597 31254 hybrid_clock.cc:648] HybridClock initialized: now 1790592589913504 us; error 83 us; skew 500 ppm
I20260928 10:49:49.914031 31254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:49 dist-test-slave-mn38 krb5kdc[31241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592589, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:49.917619 31254 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:49.919267 31254 webserver.cc:460] Webserver started at http://127.8.218.254:36597/ using document root <none> and password file <none>
I20260928 10:49:49.919996 31254 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:49.920136 31254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:49.920441 31254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:49.923803 31254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "51c697e8eb27434ebf4de1a764975a82"
format_stamp: "Formatted at 2026-09-28 10:49:49 on dist-test-slave-mn38"
server_key: "d652af2c056054e90d90b1c2ebdb20e8"
server_key_iv: "64906f2c6d102ab28f74f05a1810272b"
server_key_version: "encryptionkey@0"
I20260928 10:49:49.924700 31254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "51c697e8eb27434ebf4de1a764975a82"
format_stamp: "Formatted at 2026-09-28 10:49:49 on dist-test-slave-mn38"
server_key: "d652af2c056054e90d90b1c2ebdb20e8"
server_key_iv: "64906f2c6d102ab28f74f05a1810272b"
server_key_version: "encryptionkey@0"
I20260928 10:49:49.931324 31254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260928 10:49:49.935899 31269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:49.937650 31254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:49:49.937887 31254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "51c697e8eb27434ebf4de1a764975a82"
format_stamp: "Formatted at 2026-09-28 10:49:49 on dist-test-slave-mn38"
server_key: "d652af2c056054e90d90b1c2ebdb20e8"
server_key_iv: "64906f2c6d102ab28f74f05a1810272b"
server_key_version: "encryptionkey@0"
I20260928 10:49:49.938097 31254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:49.957808 31254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:49.962507 31254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:49.962766 31254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:49.981019 31254 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:39281
I20260928 10:49:49.981038 31320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:39281 every 8 connection(s)
I20260928 10:49:49.982501 31254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:49.986519 31321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:49.992219 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31254
I20260928 10:49:49.992400 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:49.992849 9067 external_mini_cluster.cc:1428] Setting key fc7885062f4a7ec327ba9be8c1f10ac2
I20260928 10:49:49.999856 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82: Bootstrap starting.
I20260928 10:49:50.004339 31321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:50.005811 31321 log.cc:824] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:50.009143 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:49.994973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59803 (local address 127.8.218.254:39281)
0928 10:49:49.996008 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:49.996025 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:49.996088 (+ 63us) server_negotiation.cc:407] Connection header received
0928 10:49:49.998227 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:49.998259 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:49.998821 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:49.999353 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:49.999996 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:50.001627 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:50.003797 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:50.004731 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:50.004922 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:50.006298 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:50.006307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:50.006326 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:50.007251 (+ 925us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:50.007366 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:50.007410 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:50.007541 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:50.007796 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 10:49:50.008099 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":478,"threads_started":1}
I20260928 10:49:50.009528 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82: No bootstrap required, opened a new log
I20260928 10:49:50.013293 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:50.019284 31321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c697e8eb27434ebf4de1a764975a82" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39281 } }
I20260928 10:49:50.019883 31321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:50.020026 31321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c697e8eb27434ebf4de1a764975a82, State: Initialized, Role: FOLLOWER
I20260928 10:49:50.020785 31321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c697e8eb27434ebf4de1a764975a82" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39281 } }
I20260928 10:49:50.021116 31321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:50.021277 31321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:50.021499 31321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:50.024381 31321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c697e8eb27434ebf4de1a764975a82" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39281 } }
I20260928 10:49:50.025022 31321 leader_election.cc:302] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51c697e8eb27434ebf4de1a764975a82; no voters:
I20260928 10:49:50.025868 31321 leader_election.cc:288] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:50.026042 31331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:50.026100 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:50.018830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42823 (local address 127.8.218.254:39281)
0928 10:49:50.019033 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:50.019039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:50.019459 (+ 420us) server_negotiation.cc:407] Connection header received
0928 10:49:50.019628 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:50.019633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:50.019729 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:50.019849 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:50.020406 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:50.021502 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:50.022996 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:50.023725 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:50.023868 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:50.025192 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:50.025200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:50.025204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:50.025232 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:50.025289 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:50.025313 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:50.025414 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:50.025615 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 10:49:50.025749 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 10:49:50.027366 31331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [term 1 LEADER]: Becoming Leader. State: Replica: 51c697e8eb27434ebf4de1a764975a82, State: Running, Role: LEADER
I20260928 10:49:50.028044 31331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c697e8eb27434ebf4de1a764975a82" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39281 } }
I20260928 10:49:50.028505 31321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:50.031023 31333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51c697e8eb27434ebf4de1a764975a82. Latest consensus state: current_term: 1 leader_uuid: "51c697e8eb27434ebf4de1a764975a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c697e8eb27434ebf4de1a764975a82" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39281 } } }
I20260928 10:49:50.031538 31333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:50.031035 31332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51c697e8eb27434ebf4de1a764975a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c697e8eb27434ebf4de1a764975a82" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 39281 } } }
I20260928 10:49:50.032311 31332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:50.034956 31339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:50.041141 31339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:50.052575 31339 catalog_manager.cc:1347] Generated new cluster ID: 8b929373ada144eba323489fcfd9d6d8
I20260928 10:49:50.052717 31339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:50.063014 31339 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:50.064978 31339 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:50.081130 31339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51c697e8eb27434ebf4de1a764975a82: Generated new TSK 0
I20260928 10:49:50.082423 31339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:50.094878 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31254
W20260928 10:49:50.096211 31316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59803
W20260928 10:49:50.097062 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:50.097288 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31254
2026-09-28T10:49:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:50 dist-test-slave-mn38 krb5kdc[31352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:50 dist-test-slave-mn38 krb5kdc[31352](info): set up 2 sockets
Sep 28 10:49:50 dist-test-slave-mn38 krb5kdc[31352](info): commencing operation
krb5kdc: starting...
W20260928 10:49:51.146960 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:51 dist-test-slave-mn38 krb5kdc[31352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:51.417286 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:34805
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:34805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:51.773983 31365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:51.774349 31365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:51.774489 31365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:51.782078 31365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:51.782226 31365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:51.782313 31365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:51.782392 31365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:51.782456 31365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:51.792938 31365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:34805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:51.794498 31365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:51.795965 31365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:51.809366 31370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:51.809463 31373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:51.809514 31371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:51.809880 31365 server_base.cc:1034] running on GCE node
I20260928 10:49:51.810905 31365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:51.812456 31365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:51.813687 31365 hybrid_clock.cc:648] HybridClock initialized: now 1790592591813617 us; error 64 us; skew 500 ppm
I20260928 10:49:51.814071 31365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:51 dist-test-slave-mn38 krb5kdc[31352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:51.823520 31365 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:51.825539 31365 webserver.cc:460] Webserver started at http://127.8.218.254:44269/ using document root <none> and password file <none>
I20260928 10:49:51.826370 31365 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:51.826529 31365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:51.826848 31365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:51.830202 31365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "2b620e36e4fc42b4be277d348683ab2e"
format_stamp: "Formatted at 2026-09-28 10:49:51 on dist-test-slave-mn38"
server_key: "d6ff9d067920b3369ff35664e5cb0220"
server_key_iv: "d095659b0ff1b27fbbd443f0acf75bf5"
server_key_version: "encryptionkey@0"
I20260928 10:49:51.831050 31365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "2b620e36e4fc42b4be277d348683ab2e"
format_stamp: "Formatted at 2026-09-28 10:49:51 on dist-test-slave-mn38"
server_key: "d6ff9d067920b3369ff35664e5cb0220"
server_key_iv: "d095659b0ff1b27fbbd443f0acf75bf5"
server_key_version: "encryptionkey@0"
I20260928 10:49:51.836894 31365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 10:49:51.840904 31380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:51.842592 31365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:49:51.842778 31365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "2b620e36e4fc42b4be277d348683ab2e"
format_stamp: "Formatted at 2026-09-28 10:49:51 on dist-test-slave-mn38"
server_key: "d6ff9d067920b3369ff35664e5cb0220"
server_key_iv: "d095659b0ff1b27fbbd443f0acf75bf5"
server_key_version: "encryptionkey@0"
I20260928 10:49:51.842981 31365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:51.849161 31365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:51.853685 31365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:51.853979 31365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:51.873584 31365 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:34805
I20260928 10:49:51.873593 31431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:34805 every 8 connection(s)
I20260928 10:49:51.875120 31365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:51.879412 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:51.884759 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31365
I20260928 10:49:51.884967 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:51.885548 9067 external_mini_cluster.cc:1428] Setting key fcd5b72c530a991cb5d97c4ecfe1280a
I20260928 10:49:51.893571 31432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e: Bootstrap starting.
I20260928 10:49:51.895588 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:51.887957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52651 (local address 127.8.218.254:34805)
0928 10:49:51.888977 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:51.888992 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:51.889038 (+ 46us) server_negotiation.cc:407] Connection header received
0928 10:49:51.891752 (+ 2714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:51.891787 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:51.892226 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:51.892661 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:51.893871 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:51.893878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:51.893893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:51.893915 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:51.893980 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:51.894002 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:51.894230 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:51.894368 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 10:49:51.894565 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":475,"threads_started":1}
I20260928 10:49:51.897246 31432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:51.898483 31432 log.cc:824] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:51.899888 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:51.902065 31432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e: No bootstrap required, opened a new log
I20260928 10:49:51.907272 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:51.904530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50409 (local address 127.8.218.254:34805)
0928 10:49:51.904760 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:51.904767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:51.905183 (+ 416us) server_negotiation.cc:407] Connection header received
0928 10:49:51.905389 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:51.905394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:51.905487 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:51.905578 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:51.906568 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:51.906575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:51.906579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 10:49:51.906599 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:51.906647 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 10:49:51.906665 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:51.906781 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:51.906894 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 10:49:51.906970 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 10:49:51.911113 31432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b620e36e4fc42b4be277d348683ab2e" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34805 } }
I20260928 10:49:51.911528 31432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 10:49:51.911650 31432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b620e36e4fc42b4be277d348683ab2e, State: Initialized, Role: FOLLOWER
I20260928 10:49:51.912313 31432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b620e36e4fc42b4be277d348683ab2e" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34805 } }
I20260928 10:49:51.912546 31432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 10:49:51.912657 31432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 10:49:51.912793 31432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:51.915114 31432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b620e36e4fc42b4be277d348683ab2e" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34805 } }
I20260928 10:49:51.915665 31432 leader_election.cc:302] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b620e36e4fc42b4be277d348683ab2e; no voters:
I20260928 10:49:51.916239 31432 leader_election.cc:288] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 10:49:51.916430 31442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:51.917042 31442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [term 1 LEADER]: Becoming Leader. State: Replica: 2b620e36e4fc42b4be277d348683ab2e, State: Running, Role: LEADER
I20260928 10:49:51.917637 31442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b620e36e4fc42b4be277d348683ab2e" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34805 } }
I20260928 10:49:51.918334 31432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:51.920431 31444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b620e36e4fc42b4be277d348683ab2e. Latest consensus state: current_term: 1 leader_uuid: "2b620e36e4fc42b4be277d348683ab2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b620e36e4fc42b4be277d348683ab2e" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34805 } } }
I20260928 10:49:51.920876 31444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:51.921389 31443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b620e36e4fc42b4be277d348683ab2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b620e36e4fc42b4be277d348683ab2e" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 34805 } } }
I20260928 10:49:51.921710 31443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:51.921826 31450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:51.927541 31450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:51.937974 31450 catalog_manager.cc:1347] Generated new cluster ID: 8b07b8e4c85b4963a55e7d11ccd7836c
I20260928 10:49:51.938210 31450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:51.962921 31450 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:51.963594 31450 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:51.971720 31450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b620e36e4fc42b4be277d348683ab2e: Generated new TSK 0
I20260928 10:49:51.972874 31450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:51.990288 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31365
W20260928 10:49:51.991680 31427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52651
W20260928 10:49:51.992542 9067 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31365: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 10:49:51.992693 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31365
2026-09-28T10:49:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1894 ms)
[----------] 18 tests from AuthTokenIssuingTest (34689 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:52 dist-test-slave-mn38 krb5kdc[31463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:52 dist-test-slave-mn38 krb5kdc[31463](info): set up 2 sockets
Sep 28 10:49:52 dist-test-slave-mn38 krb5kdc[31463](info): commencing operation
krb5kdc: starting...
W20260928 10:49:53.034745 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:53 dist-test-slave-mn38 krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:53.317957 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38949
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38949,127.8.218.253:33269,127.8.218.252:42367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:53.682890 31476 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:49:53.683411 31476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:53.683518 31476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:53.683629 31476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:53.691427 31476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:53.691543 31476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:53.691604 31476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:53.691677 31476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:53.691761 31476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:53.702309 31476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38949,127.8.218.253:33269,127.8.218.252:42367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:53.703755 31476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:53.705193 31476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:53.719113 31482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:53.719179 31481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:53.719148 31484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:53.720829 31476 server_base.cc:1034] running on GCE node
I20260928 10:49:53.721838 31476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:53.723475 31476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:53.724761 31476 hybrid_clock.cc:648] HybridClock initialized: now 1790592593724671 us; error 66 us; skew 500 ppm
I20260928 10:49:53.725154 31476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:53 dist-test-slave-mn38 krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592593, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:53.728742 31476 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:53.730546 31476 webserver.cc:460] Webserver started at http://127.8.218.254:43333/ using document root <none> and password file <none>
I20260928 10:49:53.731305 31476 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:53.731406 31476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:53.731681 31476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:53.735126 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "fd85b08aa2904ac98995a40152502f5c"
format_stamp: "Formatted at 2026-09-28 10:49:53 on dist-test-slave-mn38"
server_key: "e0d713f92a637ff5dec596d049871f4e"
server_key_iv: "914abde0c6d99a46d4f71470bbc34c4d"
server_key_version: "encryptionkey@0"
I20260928 10:49:53.736038 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "fd85b08aa2904ac98995a40152502f5c"
format_stamp: "Formatted at 2026-09-28 10:49:53 on dist-test-slave-mn38"
server_key: "e0d713f92a637ff5dec596d049871f4e"
server_key_iv: "914abde0c6d99a46d4f71470bbc34c4d"
server_key_version: "encryptionkey@0"
I20260928 10:49:53.742214 31476 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 10:49:53.746568 31491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:53.748224 31476 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 10:49:53.748438 31476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "fd85b08aa2904ac98995a40152502f5c"
format_stamp: "Formatted at 2026-09-28 10:49:53 on dist-test-slave-mn38"
server_key: "e0d713f92a637ff5dec596d049871f4e"
server_key_iv: "914abde0c6d99a46d4f71470bbc34c4d"
server_key_version: "encryptionkey@0"
I20260928 10:49:53.748687 31476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:53.778369 31476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:53.782992 31476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:53.783255 31476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:53.802035 31476 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:38949
I20260928 10:49:53.802060 31542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:38949 every 8 connection(s)
I20260928 10:49:53.803516 31476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:53.806936 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31476
I20260928 10:49:53.807185 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:53.808036 9067 external_mini_cluster.cc:1428] Setting key cafd39d3004955dff4efbcfa63ad3564
I20260928 10:49:53.808439 31543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:53.818100 31543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:49:53 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592593, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.253@KRBTEST.COM" created.
I20260928 10:49:53.843286 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:53.822343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38949 (local address 127.0.0.1:34512)
0928 10:49:53.823685 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:53.823728 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:53.825839 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:53.826931 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:53.826938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:53.827725 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:53.828111 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:53.828137 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:53.829866 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:53.829870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:53.831033 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:53.831050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:53.832097 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:53.832104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:53.832332 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:53.832988 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:53.833012 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:53.835415 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:53.838876 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:53.838890 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:53.838905 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:53.839340 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:53.839767 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:53.839772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:53.839776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:53.839933 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:53.840724 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:53.840733 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:53.840972 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 10:49:53.841211 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 10:49:53.841868 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"mutex_wait_us":149,"thread_start_us":919,"threads_started":1}
I20260928 10:49:53.843317 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:53.822317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.8.218.254:38949)
0928 10:49:53.823281 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:53.823296 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:53.823877 (+ 581us) server_negotiation.cc:407] Connection header received
0928 10:49:53.826363 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:53.826371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:53.826668 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:53.826889 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:53.828358 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:53.829704 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:53.831209 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:53.831839 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:53.832044 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:53.835680 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:53.835705 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:53.835720 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:53.835756 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:53.838688 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:53.839480 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:53.839487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:53.839489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:53.839641 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:53.840036 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:53.840040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:53.840042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:53.840582 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:53.840793 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:53.841350 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 10:49:53.841731 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":694,"thread_start_us":443,"threads_started":1}
I20260928 10:49:53.847095 31543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:53.848196 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:53.847751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46198)
0928 10:49:53.847945 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:53.848064 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 10:49:53.848469 31492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
I20260928 10:49:53.850265 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:53.849858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46208)
0928 10:49:53.850009 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:53.850092 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 10:49:53.850492 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:33269: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab.
Entry for principal kudu/127.8.218.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab.
I20260928 10:49:53.868786 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.253:33269
--webserver_interface=127.8.218.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab
--principal=kudu/127.8.218.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38949,127.8.218.253:33269,127.8.218.252:42367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:49:53.899901 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } attempt: 1
I20260928 10:49:53.901228 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:53.900718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46224)
0928 10:49:53.900946 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:53.901076 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 10:49:53.902449 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:53.901985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46230)
0928 10:49:53.902156 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:53.902249 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 10:49:53.902657 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:33269: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
I20260928 10:49:53.970927 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } attempt: 2
I20260928 10:49:53.972069 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:53.971600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46234)
0928 10:49:53.971826 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:53.971915 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 10:49:53.973415 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:53.972858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46244)
0928 10:49:53.973103 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:53.973263 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260928 10:49:53.973665 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:33269: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
I20260928 10:49:54.064937 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } attempt: 3
I20260928 10:49:54.066149 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.065700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46250)
0928 10:49:54.065908 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.066003 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 10:49:54.067174 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.066787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46256)
0928 10:49:54.066946 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.067021 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 10:49:54.067377 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:33269: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
I20260928 10:49:54.210642 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } attempt: 4
I20260928 10:49:54.212189 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.211535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46262)
0928 10:49:54.211925 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.212020 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260928 10:49:54.213747 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.213058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46264)
0928 10:49:54.213354 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.213526 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260928 10:49:54.213977 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:33269: Network error: Client connection negotiation failed: client connection to 127.8.218.253:33269: connect: Connection refused (error 111)
W20260928 10:49:54.228062 31550 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:49:54.228431 31550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:54.228536 31550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:54.228652 31550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:54.237154 31550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:54.237290 31550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:54.237356 31550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:54.237428 31550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:54.237557 31550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:54.248283 31550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38949,127.8.218.253:33269,127.8.218.252:42367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab
--principal=kudu/127.8.218.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.253:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.218.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:54.249734 31550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:54.251121 31550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:54.263757 31555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:54.263845 31558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:54.263757 31556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:54.264863 31550 server_base.cc:1034] running on GCE node
I20260928 10:49:54.265735 31550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:54.267192 31550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:54.268425 31550 hybrid_clock.cc:648] HybridClock initialized: now 1790592594268345 us; error 57 us; skew 500 ppm
I20260928 10:49:54.268761 31550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:54.271932 31550 init.cc:377] Logged in from keytab as kudu/127.8.218.253@KRBTEST.COM (short username kudu)
I20260928 10:49:54.273487 31550 webserver.cc:460] Webserver started at http://127.8.218.253:40429/ using document root <none> and password file <none>
I20260928 10:49:54.274197 31550 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:54.274299 31550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:54.274597 31550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:54.277660 31550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/data/instance:
uuid: "5a4c368e7ddb47cbaffe3052d9d518e4"
format_stamp: "Formatted at 2026-09-28 10:49:54 on dist-test-slave-mn38"
server_key: "109929786bfb8812dbdc7b0d6c4c1d6a"
server_key_iv: "bf8278eac6f1bf475510899b4d5e8691"
server_key_version: "encryptionkey@0"
I20260928 10:49:54.278461 31550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/wal/instance:
uuid: "5a4c368e7ddb47cbaffe3052d9d518e4"
format_stamp: "Formatted at 2026-09-28 10:49:54 on dist-test-slave-mn38"
server_key: "109929786bfb8812dbdc7b0d6c4c1d6a"
server_key_iv: "bf8278eac6f1bf475510899b4d5e8691"
server_key_version: "encryptionkey@0"
I20260928 10:49:54.283931 31550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:49:54.287912 31565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:54.289878 31550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 10:49:54.290088 31550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/wal
uuid: "5a4c368e7ddb47cbaffe3052d9d518e4"
format_stamp: "Formatted at 2026-09-28 10:49:54 on dist-test-slave-mn38"
server_key: "109929786bfb8812dbdc7b0d6c4c1d6a"
server_key_iv: "bf8278eac6f1bf475510899b4d5e8691"
server_key_version: "encryptionkey@0"
I20260928 10:49:54.290331 31550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:54.314450 31550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:54.320621 31550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:54.320972 31550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:54.341496 31550 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.253:33269
I20260928 10:49:54.341521 31616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.253:33269 every 8 connection(s)
I20260928 10:49:54.343185 31550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
I20260928 10:49:54.345566 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31550
I20260928 10:49:54.345762 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-1/wal/instance
I20260928 10:49:54.346271 9067 external_mini_cluster.cc:1428] Setting key 3ab3035241d1a238f1f6512746663740
I20260928 10:49:54.348129 31617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:54.357496 31617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.218.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.252@KRBTEST.COM" created.
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:54.384956 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.361005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.8.218.254:38949)
0928 10:49:54.361938 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.361944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.363461 (+ 1517us) server_negotiation.cc:407] Connection header received
0928 10:49:54.366002 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.366009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.366117 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.366228 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.368185 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.369027 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.370895 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.371342 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.371464 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.377940 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.377949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.377953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.378016 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.381395 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.382377 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.382383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.382386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.382543 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.382980 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.382984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.382987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.383364 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.383534 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.384335 (+ 801us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.384575 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":577,"threads_started":1}
I20260928 10:49:54.385746 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.362042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38949 (local address 127.0.0.1:34528)
0928 10:49:54.363197 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.363245 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:54.365595 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:54.366563 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:54.366574 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:54.367221 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:54.367913 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.367936 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.369243 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.369248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.370716 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.370733 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.371604 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.371613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.371854 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.372519 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:54.372548 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:54.377603 (+ 5055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:54.381652 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.381663 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.381681 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.382213 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.382658 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.382662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.382664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.382864 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.383496 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:54.383502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:54.383897 (+ 395us) client_negotiation.cc:769] Sending connection context
0928 10:49:54.384139 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 10:49:54.384550 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":496,"threads_started":1}
I20260928 10:49:54.387838 31617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
I20260928 10:49:54.404371 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.388717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46272)
0928 10:49:54.388931 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.388951 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:54.389094 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:54.390359 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:54.390363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:54.390780 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:54.390930 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.390939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.392782 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.392786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.393739 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.393755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.394933 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.394939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.395096 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.395731 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:54.395751 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:54.398193 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:54.401706 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.401712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.401716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.402273 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.402672 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.402678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.402682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.402862 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.403479 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:54.403483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:54.403608 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 10:49:54.403769 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 10:49:54.404072 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 10:49:54.404526 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.388991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.8.218.253:33269)
0928 10:49:54.389828 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.389836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.389868 (+ 32us) server_negotiation.cc:407] Connection header received
0928 10:49:54.389927 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.389934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.390185 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.390334 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.391103 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.392628 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.393867 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.394711 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.394843 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.398484 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.398501 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.398517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.398553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.401538 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.402418 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.402425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.402427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.402550 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.402963 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.402966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.402968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.403342 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.403516 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.403884 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.404208 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":658,"thread_start_us":495,"threads_started":1}
I20260928 10:49:54.407133 31617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:54.408501 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.407990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59392)
0928 10:49:54.408183 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.408326 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 10:49:54.408829 31568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
I20260928 10:49:54.410755 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.410278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59406)
0928 10:49:54.410449 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.410613 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 10:49:54.411083 31617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab.
Entry for principal kudu/127.8.218.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab.
I20260928 10:49:54.413877 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.252:42367
--webserver_interface=127.8.218.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab
--principal=kudu/127.8.218.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:38949,127.8.218.253:33269,127.8.218.252:42367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:49:54.460428 31617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 1
I20260928 10:49:54.461684 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.461076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59418)
0928 10:49:54.461316 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.461486 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 10:49:54.462800 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.462398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59420)
0928 10:49:54.462582 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.462677 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 10:49:54.463019 31617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
I20260928 10:49:54.513340 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } attempt: 5
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592593, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
I20260928 10:49:54.527990 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.514128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46276)
0928 10:49:54.514362 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.514386 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:54.514533 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:54.514955 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:54.514962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:54.515393 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:54.515586 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.515596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.516643 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.516649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.517652 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.517669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.518457 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.518463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.518678 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.519408 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:54.519428 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:54.521824 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:54.525267 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.525275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.525280 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.525789 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.526222 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.526227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.526232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.526400 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.526958 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:54.526963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:54.527135 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 10:49:54.527288 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 10:49:54.527608 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 10:49:54.527985 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.514194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.8.218.253:33269)
0928 10:49:54.514388 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.514394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.514453 (+ 59us) server_negotiation.cc:407] Connection header received
0928 10:49:54.514710 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.514715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.514793 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.514885 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.515753 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.516483 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.517823 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.518279 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.518402 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.522081 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.522087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.522091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.522133 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.525050 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.525935 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.525940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.525942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.526098 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.526525 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.526529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.526531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.526840 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.526979 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.527437 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.527667 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 10:49:54.529412 31543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:54.530741 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.530205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59422)
0928 10:49:54.530455 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.530555 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 10:49:54.531273 31617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 2
I20260928 10:49:54.531881 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.531485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59430)
0928 10:49:54.531632 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.531747 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 10:49:54.532115 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
I20260928 10:49:54.532495 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.531742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59438)
0928 10:49:54.531939 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.532335 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 10:49:54.533560 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.533111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59452)
0928 10:49:54.533321 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.533393 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 10:49:54.533802 31617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
I20260928 10:49:54.583413 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 1
I20260928 10:49:54.584712 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.584151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59456)
0928 10:49:54.584393 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.584505 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 10:49:54.585769 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.585371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59460)
0928 10:49:54.585552 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.585637 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 10:49:54.585975 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
I20260928 10:49:54.625098 31617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 3
I20260928 10:49:54.626278 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.625801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59466)
0928 10:49:54.626009 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.626104 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 10:49:54.627801 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.627197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59468)
0928 10:49:54.627513 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.627606 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
W20260928 10:49:54.628088 31617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
I20260928 10:49:54.654207 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 2
I20260928 10:49:54.655676 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.654991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59482)
0928 10:49:54.655325 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.655452 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 10:49:54.657150 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.656570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59494)
0928 10:49:54.656844 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.656958 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260928 10:49:54.657413 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
I20260928 10:49:54.763700 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 3
I20260928 10:49:54.764897 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.764370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59508)
0928 10:49:54.764585 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.764695 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 10:49:54.765955 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.765544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59516)
0928 10:49:54.765737 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.765826 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260928 10:49:54.766148 31543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
W20260928 10:49:54.769541 31625 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:49:54.769874 31625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:54.769946 31625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:54.770026 31625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 10:49:54.771303 31617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 4
I20260928 10:49:54.772666 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.771988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59524)
0928 10:49:54.772311 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.772483 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 10:49:54.773743 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.773333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59526)
0928 10:49:54.773510 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.773601 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 10:49:54.773938 31617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:42367: Network error: Client connection negotiation failed: client connection to 127.8.218.252:42367: connect: Connection refused (error 111)
W20260928 10:49:54.778080 31625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:54.778185 31625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:54.778278 31625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:54.778344 31625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:54.778427 31625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:54.789974 31625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:38949,127.8.218.253:33269,127.8.218.252:42367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab
--principal=kudu/127.8.218.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.252:42367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.218.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:54.791812 31625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:54.793742 31625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:54.808188 31630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:54.808306 31633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:54.809108 31631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:54.809581 31625 server_base.cc:1034] running on GCE node
I20260928 10:49:54.810678 31625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:54.812521 31625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:54.813838 31625 hybrid_clock.cc:648] HybridClock initialized: now 1790592594813767 us; error 64 us; skew 500 ppm
I20260928 10:49:54.814249 31625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:54.818151 31625 init.cc:377] Logged in from keytab as kudu/127.8.218.252@KRBTEST.COM (short username kudu)
I20260928 10:49:54.820115 31625 webserver.cc:460] Webserver started at http://127.8.218.252:33563/ using document root <none> and password file <none>
I20260928 10:49:54.820948 31625 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:54.821076 31625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:54.821455 31625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:54.824923 31625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/data/instance:
uuid: "1080a27ba49d46fbb09dd1a531a5ea13"
format_stamp: "Formatted at 2026-09-28 10:49:54 on dist-test-slave-mn38"
server_key: "9bddb0881535fca0416cddbf8ee96446"
server_key_iv: "fd297fc1338f13b78ab993a01572f5e1"
server_key_version: "encryptionkey@0"
I20260928 10:49:54.825958 31625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/wal/instance:
uuid: "1080a27ba49d46fbb09dd1a531a5ea13"
format_stamp: "Formatted at 2026-09-28 10:49:54 on dist-test-slave-mn38"
server_key: "9bddb0881535fca0416cddbf8ee96446"
server_key_iv: "fd297fc1338f13b78ab993a01572f5e1"
server_key_version: "encryptionkey@0"
I20260928 10:49:54.833174 31625 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 10:49:54.837807 31640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:54.839630 31625 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 10:49:54.839864 31625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/wal
uuid: "1080a27ba49d46fbb09dd1a531a5ea13"
format_stamp: "Formatted at 2026-09-28 10:49:54 on dist-test-slave-mn38"
server_key: "9bddb0881535fca0416cddbf8ee96446"
server_key_iv: "fd297fc1338f13b78ab993a01572f5e1"
server_key_version: "encryptionkey@0"
I20260928 10:49:54.840111 31625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:54.861037 31625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:54.865432 31625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:54.865729 31625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:54.884459 31625 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.252:42367
I20260928 10:49:54.884485 31691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.252:42367 every 8 connection(s)
I20260928 10:49:54.885977 31625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
I20260928 10:49:54.890177 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:54.890723 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31625
I20260928 10:49:54.890921 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790592230404718-9067-0/minicluster-data/master-2/wal/instance
I20260928 10:49:54.891464 9067 external_mini_cluster.cc:1428] Setting key b1f79aa23f1fd68a6b46f795a4c34e6c
I20260928 10:49:54.901010 31692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:54.918672 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.893883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.8.218.252:42367)
0928 10:49:54.894798 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.894815 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.894861 (+ 46us) server_negotiation.cc:407] Connection header received
0928 10:49:54.897428 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.897498 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.898204 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.898776 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.900367 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.902497 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.904297 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.905386 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.905675 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.910239 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.910253 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.910266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.910326 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.914196 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.915056 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.915067 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.915077 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.915337 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.915801 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.915805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.915807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.916393 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.916628 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.917121 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.917581 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":391,"threads_started":1}
I20260928 10:49:54.921746 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.904813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:38949 (local address 127.0.0.1:34542)
0928 10:49:54.905856 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.905908 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:54.906064 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:54.906739 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:54.906747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:54.907435 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:54.907698 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.907709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.908768 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.908772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.910380 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.910395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.911247 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.911255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.911477 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.912212 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:54.912241 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:54.915272 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:54.918708 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.918720 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.918726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.919236 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.919780 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.919784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.919786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.919939 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.920542 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:54.920549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:54.920763 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 10:49:54.920971 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 10:49:54.921304 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":579,"threads_started":1}
I20260928 10:49:54.921764 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.904797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.8.218.254:38949)
0928 10:49:54.905423 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.905429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.905995 (+ 566us) server_negotiation.cc:407] Connection header received
0928 10:49:54.906452 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.906457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.906532 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.906643 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.907867 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.908583 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.910542 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.911013 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.911146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.915511 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.915519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.915524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.915579 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.918550 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.919396 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.919404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.919407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.919627 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.920066 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.920071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.920075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.920396 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.920520 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.921116 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.921371 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":320,"threads_started":1}
I20260928 10:49:54.923281 31692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:54.936780 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.924033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:33269 (local address 127.0.0.1:46278)
0928 10:49:54.924227 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.924246 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:54.924359 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:54.924650 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:54.924653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:54.924989 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:54.925130 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.925138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.926258 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.926264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.927328 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.927341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.928087 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.928093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.928274 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.928764 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:54.928779 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:54.931024 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:54.934309 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.934316 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.934320 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.934730 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.935135 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.935141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.935145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.935296 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.935814 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:54.935818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:54.935972 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 10:49:54.936094 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 10:49:54.936420 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 10:49:54.937220 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.922498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.8.218.254:38949)
0928 10:49:54.922647 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.922652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.922699 (+ 47us) server_negotiation.cc:407] Connection header received
0928 10:49:54.922905 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.922909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.922976 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.923066 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.924392 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.925295 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.926817 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.927294 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.927396 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.931732 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.931737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.931741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.931775 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.934600 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.935306 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.935310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.935312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.935444 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.935811 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.935815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.935817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.936091 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.936419 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.936704 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.936942 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 10:49:54.936810 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.924073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.8.218.253:33269)
0928 10:49:54.924251 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.924257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.924375 (+ 118us) server_negotiation.cc:407] Connection header received
0928 10:49:54.924444 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.924449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.924526 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.924632 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.925303 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.926073 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.927476 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.927940 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.928046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.931228 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.931234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.931237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.931272 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.934147 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.934858 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.934863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.934865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.935010 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.935403 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.935407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.935409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.935695 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.935837 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.936213 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.936434 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 10:49:54.938196 31692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:54.943368 31543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 4
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
Sep 28 10:49:54 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592593, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
I20260928 10:49:54.954790 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.939016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59546)
0928 10:49:54.939157 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.939171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:54.939277 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:54.939599 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:54.939604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:54.940004 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:54.940161 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.940170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.941526 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.941531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.942913 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.942924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.944035 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.944041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.944282 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.944979 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:54.945002 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:54.947963 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:54.951677 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.951683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.951688 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.952257 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.952683 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.952688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.952692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.952850 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.953503 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:54.953508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:54.953689 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 10:49:54.953839 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 10:49:54.954199 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 10:49:54.955049 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.938919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.8.218.252:42367)
0928 10:49:54.939090 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.939096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.939225 (+ 129us) server_negotiation.cc:407] Connection header received
0928 10:49:54.939374 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.939379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.939446 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.939533 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.940302 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.941349 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.943061 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.943698 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.943829 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.948160 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.948168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.948173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.948220 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.951516 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.952410 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.952416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.952419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.952556 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.952970 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.952974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.952976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.953309 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.953540 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.953945 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.954476 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":48}
I20260928 10:49:54.959014 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.939647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.8.218.253:33269)
0928 10:49:54.939806 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.939810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.939823 (+ 13us) server_negotiation.cc:407] Connection header received
0928 10:49:54.939865 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.939868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.939922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.940000 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.941548 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.942612 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.944454 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.945056 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.945276 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.950564 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.950572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.950578 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.950624 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.953907 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.954911 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.954916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.954919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.955098 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.955739 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.955745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.955748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.956140 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.956438 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.958238 (+ 1800us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.958542 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 10:49:54.963820 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.944082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.8.218.252:42367)
0928 10:49:54.945339 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:54.945344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:54.945365 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:49:54.945413 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:54.945418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:54.945525 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:54.945633 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:54.946517 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.947372 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.949136 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.949873 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.950008 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.953771 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:54.953779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:54.953783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:54.953823 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:54.959301 (+ 5478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.960131 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.960138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.960141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.960470 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:54.961201 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:54.961207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:54.961210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:54.961659 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:54.961910 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:54.963139 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0928 10:49:54.963443 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":877,"threads_started":1}
I20260928 10:49:54.964372 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:54.944037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59554)
0928 10:49:54.944529 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:54.944553 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:54.944677 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:54.945704 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:54.945710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:54.946159 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:54.946377 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.946392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.947512 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.947518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.948691 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:54.948721 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:54.950070 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:54.950079 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:54.950306 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:54.950877 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:54.950900 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:54.953475 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:54.959522 (+ 6047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.959529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.959533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.959991 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.960695 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:54.960701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:54.960705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:54.960970 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:54.962141 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:54.962148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:54.962586 (+ 438us) client_negotiation.cc:769] Sending connection context
0928 10:49:54.962774 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 10:49:54.963166 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353}
I20260928 10:49:54.964934 31692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13: Bootstrap starting.
I20260928 10:49:54.965741 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:54.970238 31692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:54.968643 31518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fd85b08aa2904ac98995a40152502f5c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34546
I20260928 10:49:54.971892 31543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c: Bootstrap starting.
I20260928 10:49:54.971936 31692 log.cc:824] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:54.976137 31692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13: No bootstrap required, opened a new log
I20260928 10:49:54.976498 31543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:54.978092 31543 log.cc:824] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:54.982044 31543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c: No bootstrap required, opened a new log
I20260928 10:49:54.986878 31692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } }
I20260928 10:49:54.987382 31692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1080a27ba49d46fbb09dd1a531a5ea13, State: Initialized, Role: FOLLOWER
I20260928 10:49:54.987954 31692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } }
I20260928 10:49:54.989022 31701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:54.989586 31701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:54.989718 31543 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } }
I20260928 10:49:54.990170 31543 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd85b08aa2904ac98995a40152502f5c, State: Initialized, Role: FOLLOWER
I20260928 10:49:54.990530 31692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:54.990715 31543 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } }
I20260928 10:49:54.991814 31702 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:54.992321 31702 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:54.993345 31543 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:54.993402 31518 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 10:49:54.993700 31518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:54.997020 31518 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } }
I20260928 10:49:54.999986 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd85b08aa2904ac98995a40152502f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1080a27ba49d46fbb09dd1a531a5ea13"
I20260928 10:49:55.000564 31667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:55.001674 31518 leader_election.cc:288] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [CANDIDATE]: Term 1 election: Requested vote from peers 5a4c368e7ddb47cbaffe3052d9d518e4 (127.8.218.253:33269), 1080a27ba49d46fbb09dd1a531a5ea13 (127.8.218.252:42367)
I20260928 10:49:55.005923 31667 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd85b08aa2904ac98995a40152502f5c in term 1.
I20260928 10:49:55.006825 31494 leader_election.cc:302] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1080a27ba49d46fbb09dd1a531a5ea13, fd85b08aa2904ac98995a40152502f5c; no voters:
I20260928 10:49:55.007440 31702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:55.015789 31702 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [term 1 LEADER]: Becoming Leader. State: Replica: fd85b08aa2904ac98995a40152502f5c, State: Running, Role: LEADER
I20260928 10:49:55.017227 31702 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } }
I20260928 10:49:55.034535 31729 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 10:49:55.036420 31497 tablet.cc:2369] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:55.038143 31625 master_runner.cc:186] Error getting master registration for 127.8.218.254:38949: OK, instance_id { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" instance_seqno: 1790592593796437 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:49:55.048970 31625 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:49:55.055058 31717 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd85b08aa2904ac98995a40152502f5c. Latest consensus state: current_term: 1 leader_uuid: "fd85b08aa2904ac98995a40152502f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:55.055575 31717 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:55.057070 31733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:55.073081 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.032466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.8.218.252:42367)
0928 10:49:55.032651 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.032658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.047920 (+ 15262us) server_negotiation.cc:407] Connection header received
0928 10:49:55.051332 (+ 3412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.051341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.051496 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.051585 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.059379 (+ 7794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.060430 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.062493 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.063186 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.063298 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.066508 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:55.066515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:55.066520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:55.066566 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:55.069951 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:55.070758 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:55.070765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:55.070769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:55.070914 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:55.071353 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:55.071360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:55.071363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:55.071733 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:55.071967 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.072513 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.072743 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:49:55.073127 31617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } attempt: 5
I20260928 10:49:55.075505 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.031954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.8.218.253:33269)
0928 10:49:55.032183 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.032191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.049823 (+ 17632us) server_negotiation.cc:407] Connection header received
0928 10:49:55.049999 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.050005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.050097 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.050215 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.052605 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.054285 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.056421 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.057084 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.057201 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.063840 (+ 6639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:55.063849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:55.063856 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:55.063917 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:55.068235 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:55.072429 (+ 4194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:55.072436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:55.072440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:55.072727 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:55.073418 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:55.073424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:55.073428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:55.073986 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:55.074244 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.074929 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.075173 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:49:55.076829 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.027241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.8.218.254:38949)
0928 10:49:55.027461 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.027469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.046730 (+ 19261us) server_negotiation.cc:407] Connection header received
0928 10:49:55.048064 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.048072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.048180 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.048275 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.057609 (+ 9334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.058766 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.064382 (+ 5616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.065062 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.065176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.068188 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:55.068195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:55.068205 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:55.068260 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:55.071849 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:55.072980 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:55.072986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:55.072989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:55.074553 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:55.075005 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:55.075010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:55.075013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:55.075418 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:55.075667 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.076202 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.076462 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 10:49:55.077574 31733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:55.088107 31667 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [term 1 FOLLOWER]: Refusing update from remote peer fd85b08aa2904ac98995a40152502f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:55.089661 31702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 28 10:49:55 dist-test-slave-mn38 krb5kdc[31463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
I20260928 10:49:55.082268 31476 master_runner.cc:186] Error getting master registration for 127.8.218.253:33269: OK, instance_id { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" instance_seqno: 1790592594335825 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:49:55.100522 31476 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 10:49:55.106890 31701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd85b08aa2904ac98995a40152502f5c. Latest consensus state: current_term: 1 leader_uuid: "fd85b08aa2904ac98995a40152502f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:55.107261 31701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:55.107858 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.073706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:42367 (local address 127.0.0.1:59558)
0928 10:49:55.074200 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:55.074219 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:55.074349 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:55.076241 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:55.076246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:55.076734 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:55.076962 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:55.076975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.079327 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.079333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:55.080708 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:55.080727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.083013 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.083019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:55.083275 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.089658 (+ 6383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:55.089689 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:55.092879 (+ 3190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:55.104006 (+ 11127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:55.104014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:55.104019 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:55.104715 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:55.105663 (+ 948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:55.105669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:55.105673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:55.105894 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:55.106715 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:55.106722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:55.106919 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 10:49:55.107075 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 10:49:55.107441 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367}
I20260928 10:49:55.110539 31717 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd85b08aa2904ac98995a40152502f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:55.110859 31717 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:55.112027 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.074018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.8.218.252:42367)
0928 10:49:55.075904 (+ 1886us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.075909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.075926 (+ 17us) server_negotiation.cc:407] Connection header received
0928 10:49:55.075975 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.075980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.076053 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.077153 (+ 1100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.077376 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.079099 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.082195 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.082838 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.083623 (+ 785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.093224 (+ 9601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:55.093233 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:55.093238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:55.093299 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:55.103708 (+ 10409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:55.104979 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:55.104985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:55.104989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:55.105190 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:55.106067 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:55.106072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:55.106075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:55.106551 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:55.111157 (+ 4606us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.111345 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.111629 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1756}
I20260928 10:49:55.112450 31701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd85b08aa2904ac98995a40152502f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:55.112710 31701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:55.116824 31733 catalog_manager.cc:1347] Generated new cluster ID: 26350450478143b0811085e22ab3a806
I20260928 10:49:55.116964 31733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:55.118063 31617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4: Bootstrap starting.
I20260928 10:49:55.123225 31617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:55.124608 31617 log.cc:824] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:55.136428 31617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4: No bootstrap required, opened a new log
I20260928 10:49:55.138458 31733 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:55.140386 31733 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:55.149710 31617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } }
I20260928 10:49:55.150103 31617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4c368e7ddb47cbaffe3052d9d518e4, State: Initialized, Role: FOLLOWER
I20260928 10:49:55.150964 31617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } }
I20260928 10:49:55.152053 31742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:55.153795 31742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:55.153828 31617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:55.155269 31591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:55.155686 31733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c: Generated new TSK 0
I20260928 10:49:55.157054 31733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 10:49:55.159907 31591 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [term 1 FOLLOWER]: Refusing update from remote peer fd85b08aa2904ac98995a40152502f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:55.161587 31717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:55.164644 31592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd85b08aa2904ac98995a40152502f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4"
I20260928 10:49:55.215096 31742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd85b08aa2904ac98995a40152502f5c. Latest consensus state: current_term: 1 leader_uuid: "fd85b08aa2904ac98995a40152502f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:55.215480 31742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:55.216014 31753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd85b08aa2904ac98995a40152502f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:55.216602 31753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:55.217350 31592 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fd85b08aa2904ac98995a40152502f5c for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260928 10:49:55.232944 31760 catalog_manager.cc:1259] Loaded cluster ID: 26350450478143b0811085e22ab3a806
I20260928 10:49:55.233210 31760 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4: loading cluster ID for follower catalog manager: success
I20260928 10:49:55.237555 31760 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4: acquiring CA information for follower catalog manager: success
I20260928 10:49:55.240700 31760 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a4c368e7ddb47cbaffe3052d9d518e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 10:49:55.283748 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.256842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.8.218.252:42367)
0928 10:49:55.257135 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.257142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.281205 (+ 24063us) server_negotiation.cc:407] Connection header received
0928 10:49:55.281824 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.281833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.281933 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.282011 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.283441 (+ 1430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59562: BlockingRecv error: recv got EOF from 127.0.0.1:59562 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
W20260928 10:49:55.286757 31623 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.256916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.8.218.253:33269)
0928 10:49:55.257328 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.257334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.284785 (+ 27451us) server_negotiation.cc:407] Connection header received
0928 10:49:55.284921 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.284927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.285008 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.285111 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.286512 (+ 1401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46304: BlockingRecv error: recv got EOF from 127.0.0.1:46304 (error 108)
Metrics: {"server-negotiator.queue_time_us":250}
W20260928 10:49:55.287914 31696 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.256176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.8.218.254:38949)
0928 10:49:55.256428 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.256434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.284576 (+ 28142us) server_negotiation.cc:407] Connection header received
0928 10:49:55.285342 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.285348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.285458 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.285542 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.287655 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34570: BlockingRecv error: recv got EOF from 127.0.0.1:34570 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:49:55.299930 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.293140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.8.218.253:33269)
0928 10:49:55.293345 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.293353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.293917 (+ 564us) server_negotiation.cc:407] Connection header received
0928 10:49:55.294260 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.294267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.294519 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.294704 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.295149 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.296067 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.297487 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.297945 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.298074 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.298607 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.299207 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.299369 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.299464 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.299578 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 10:49:55.315692 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.307141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.8.218.252:42367)
0928 10:49:55.307464 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.307472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.314156 (+ 6684us) server_negotiation.cc:407] Connection header received
0928 10:49:55.314393 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.314398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.314508 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.314604 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.315435 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59568: BlockingRecv error: recv got EOF from 127.0.0.1:59568 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
I20260928 10:49:55.327334 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.320813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.8.218.253:33269)
0928 10:49:55.321031 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.321038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.321154 (+ 116us) server_negotiation.cc:407] Connection header received
0928 10:49:55.321457 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.321463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.321585 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.321670 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.322127 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.322870 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.324758 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.325516 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.325639 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.326385 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.326562 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.326629 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.326896 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.326998 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 10:49:55.339627 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.335712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.8.218.252:42367)
0928 10:49:55.335950 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.335958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.338115 (+ 2157us) server_negotiation.cc:407] Connection header received
0928 10:49:55.338323 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.338329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.338412 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.338500 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.339375 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59576: BlockingRecv error: recv got EOF from 127.0.0.1:59576 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 10:49:55.394161 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.387082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.8.218.253:33269)
0928 10:49:55.387297 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.387305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.387863 (+ 558us) server_negotiation.cc:407] Connection header received
0928 10:49:55.388061 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.388066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.388142 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.388221 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.388689 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.389681 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.391697 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.392156 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.392325 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.393115 (+ 790us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.393267 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.393386 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.393636 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.393740 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 10:49:55.408039 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.403812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.8.218.252:42367)
0928 10:49:55.404033 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.404040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.406364 (+ 2324us) server_negotiation.cc:407] Connection header received
0928 10:49:55.406630 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.406637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.406766 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.406863 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.407753 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59588: BlockingRecv error: recv got EOF from 127.0.0.1:59588 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 10:49:55.422029 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.415622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.8.218.253:33269)
0928 10:49:55.415902 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.415909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.416446 (+ 537us) server_negotiation.cc:407] Connection header received
0928 10:49:55.416675 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.416681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.416776 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.416856 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.417315 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.418162 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.419628 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.420319 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.420476 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.421184 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.421326 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.421413 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.421615 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.421717 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260928 10:49:55.430799 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.428369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.8.218.252:42367)
0928 10:49:55.428691 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.428699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.429286 (+ 587us) server_negotiation.cc:407] Connection header received
0928 10:49:55.429510 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.429517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.429610 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.429699 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.430550 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59600: BlockingRecv error: recv got EOF from 127.0.0.1:59600 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
I20260928 10:49:55.449158 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.443311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.8.218.253:33269)
0928 10:49:55.443532 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.443540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.444058 (+ 518us) server_negotiation.cc:407] Connection header received
0928 10:49:55.444281 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.444288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.444372 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.444472 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.444894 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.445797 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.447222 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.447722 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.447844 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.448307 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.448429 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.448504 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.448659 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.448750 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 10:49:55.459489 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.456789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.8.218.252:42367)
0928 10:49:55.457022 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.457030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.457942 (+ 912us) server_negotiation.cc:407] Connection header received
0928 10:49:55.458133 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.458144 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.458222 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.458309 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.459233 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59602: BlockingRecv error: recv got EOF from 127.0.0.1:59602 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:49:55.486994 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.481430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.8.218.253:33269)
0928 10:49:55.481682 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.481689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.482191 (+ 502us) server_negotiation.cc:407] Connection header received
0928 10:49:55.482391 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.482396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.482467 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.482541 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.482992 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.483806 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.485192 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.485696 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.485819 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.486241 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.486388 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.486466 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.486615 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.486716 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 10:49:55.500425 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.498114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.8.218.252:42367)
0928 10:49:55.498331 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.498339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.498982 (+ 643us) server_negotiation.cc:407] Connection header received
0928 10:49:55.499203 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.499209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.499298 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.499385 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.500176 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59604: BlockingRecv error: recv got EOF from 127.0.0.1:59604 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:49:55.542514 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.536129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.8.218.253:33269)
0928 10:49:55.536431 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.536439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.537018 (+ 579us) server_negotiation.cc:407] Connection header received
0928 10:49:55.537249 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.537254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.537337 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.537429 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.537925 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.538742 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.540438 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.540959 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.541083 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.541651 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.541779 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.541872 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.542019 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.542119 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 10:49:55.561501 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.558928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.8.218.252:42367)
0928 10:49:55.559151 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.559158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.559774 (+ 616us) server_negotiation.cc:407] Connection header received
0928 10:49:55.559961 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.559968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.560051 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.560137 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.561273 (+ 1136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59606: BlockingRecv error: recv got EOF from 127.0.0.1:59606 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:49:55.594516 31717 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd85b08aa2904ac98995a40152502f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd85b08aa2904ac98995a40152502f5c" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 38949 } } peers { permanent_uuid: "5a4c368e7ddb47cbaffe3052d9d518e4" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 33269 } } peers { permanent_uuid: "1080a27ba49d46fbb09dd1a531a5ea13" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 42367 } } }
I20260928 10:49:55.594942 31717 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd85b08aa2904ac98995a40152502f5c [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:55.639807 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.632400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.8.218.253:33269)
0928 10:49:55.632591 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.632598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.633195 (+ 597us) server_negotiation.cc:407] Connection header received
0928 10:49:55.633392 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.633398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.633538 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.633666 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.634139 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.635436 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.637566 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.638263 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.638420 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.638957 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.639126 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.639237 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.639381 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.639478 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 10:49:55.649969 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.647467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.8.218.252:42367)
0928 10:49:55.647658 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.647664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.648228 (+ 564us) server_negotiation.cc:407] Connection header received
0928 10:49:55.648441 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.648447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.648560 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.648676 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.649720 (+ 1044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59616: BlockingRecv error: recv got EOF from 127.0.0.1:59616 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 10:49:55.786696 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:55.781558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.8.218.253:33269)
0928 10:49:55.781796 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.781802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.782049 (+ 247us) server_negotiation.cc:407] Connection header received
0928 10:49:55.782266 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.782271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.782346 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.782434 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:55.782905 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.783653 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.785006 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:55.785455 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:55.785552 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:55.785954 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:55.786071 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:55.786133 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:55.786281 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 10:49:55.786364 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 10:49:55.795425 31698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:55.793141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.8.218.252:42367)
0928 10:49:55.793376 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:55.793380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:55.793711 (+ 331us) server_negotiation.cc:407] Connection header received
0928 10:49:55.794016 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:55.794021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:55.794205 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:55.794308 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:55.795193 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59628: BlockingRecv error: recv got EOF from 127.0.0.1:59628 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 10:49:56.039444 31729 catalog_manager.cc:1259] Loaded cluster ID: 26350450478143b0811085e22ab3a806
I20260928 10:49:56.039579 31729 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13: loading cluster ID for follower catalog manager: success
I20260928 10:49:56.042621 31729 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13: acquiring CA information for follower catalog manager: success
I20260928 10:49:56.044900 31729 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1080a27ba49d46fbb09dd1a531a5ea13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:49:56.061774 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:56.055111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.8.218.253:33269)
0928 10:49:56.055368 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:56.055374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:56.055824 (+ 450us) server_negotiation.cc:407] Connection header received
0928 10:49:56.056081 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:56.056086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:56.056170 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:56.056257 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:56.056918 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.057760 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.059841 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.060279 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.060386 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:56.061050 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:56.061191 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:56.061260 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:56.061418 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 10:49:56.061529 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 10:49:56.075011 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:56.067995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.8.218.252:42367)
0928 10:49:56.068194 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:56.068200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:56.068464 (+ 264us) server_negotiation.cc:407] Connection header received
0928 10:49:56.068625 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:56.068631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:56.068895 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:56.069036 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:56.069480 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.070632 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.072089 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.072799 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.072942 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:56.073531 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:56.074307 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:56.074476 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:56.074539 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 10:49:56.074642 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 10:49:56.090418 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:56.081907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.8.218.252:42367)
0928 10:49:56.082088 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:56.082094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:56.083280 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 10:49:56.083562 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:56.083569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:56.083655 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:56.083730 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:56.084470 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.085638 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.087945 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.088652 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.088758 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:56.089480 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:56.089654 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:56.089735 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:56.089967 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 10:49:56.090064 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 10:49:56.090809 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:56.081971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.8.218.253:33269)
0928 10:49:56.082237 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:56.082242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:56.083977 (+ 1735us) server_negotiation.cc:407] Connection header received
0928 10:49:56.084183 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:56.084188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:56.084272 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:56.084371 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:56.085088 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.086126 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.088633 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.089293 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.089388 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:56.090027 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:56.090160 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:56.090234 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:56.090452 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 10:49:56.090541 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260928 10:49:56.091850 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:56.082173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.8.218.254:38949)
0928 10:49:56.082904 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:56.082910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:56.082932 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:49:56.083048 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:56.083054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:56.083269 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:56.084804 (+ 1535us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:49:56.085010 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.086144 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.088703 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:56.089491 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:56.090146 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:56.090367 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:49:56.090920 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:49:56.091083 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:56.091425 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 10:49:56.091529 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":491,"threads_started":1}
I20260928 10:49:56.097622 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31476
I20260928 10:49:56.098876 31538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:56.214159 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31476
I20260928 10:49:56.234041 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31550
I20260928 10:49:56.235399 31612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:56.347687 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31550
I20260928 10:49:56.365408 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31625
I20260928 10:49:56.366688 31687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:49:56.487214 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31625
2026-09-28T10:49:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4499 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 10:49:56 dist-test-slave-mn38 krb5kdc[31884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 10:49:56 dist-test-slave-mn38 krb5kdc[31884](info): set up 2 sockets
Sep 28 10:49:56 dist-test-slave-mn38 krb5kdc[31884](info): commencing operation
krb5kdc: starting...
W20260928 10:49:57.526855 9067 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/test-user.keytab.
Sep 28 10:49:57 dist-test-slave-mn38 krb5kdc[31884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T10:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T10:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.218.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
Entry for principal kudu/127.8.218.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab.
I20260928 10:49:57.793380 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:44301
--webserver_interface=127.8.218.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:44301,127.8.218.253:45801,127.8.218.252:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 10:49:58.149655 31897 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:49:58.150033 31897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:58.150131 31897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:58.150251 31897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:58.158289 31897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:58.158413 31897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:58.158497 31897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:58.158565 31897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:58.158650 31897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:58.169162 31897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:44301,127.8.218.253:45801,127.8.218.252:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.254.keytab
--principal=kudu/127.8.218.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.254:44301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.218.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:58.170547 31897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:58.171871 31897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:58.183835 31905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:58.183867 31902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:58.183900 31903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:58.184644 31897 server_base.cc:1034] running on GCE node
I20260928 10:49:58.185513 31897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:58.187371 31897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:58.188671 31897 hybrid_clock.cc:648] HybridClock initialized: now 1790592598188577 us; error 57 us; skew 500 ppm
I20260928 10:49:58.189098 31897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:58 dist-test-slave-mn38 krb5kdc[31884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:58.192893 31897 init.cc:377] Logged in from keytab as kudu/127.8.218.254@KRBTEST.COM (short username kudu)
I20260928 10:49:58.194875 31897 webserver.cc:460] Webserver started at http://127.8.218.254:40263/ using document root <none> and password file <none>
I20260928 10:49:58.195725 31897 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:58.195854 31897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:58.196214 31897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:58.199775 31897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/data/instance:
uuid: "50af76c70c884bb79ee941714462d551"
format_stamp: "Formatted at 2026-09-28 10:49:58 on dist-test-slave-mn38"
server_key: "e998baefae0f5468961ca72fa4b1ea9a"
server_key_iv: "80f86d15ce65811bc714f7839a6dade8"
server_key_version: "encryptionkey@0"
I20260928 10:49:58.200793 31897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/wal/instance:
uuid: "50af76c70c884bb79ee941714462d551"
format_stamp: "Formatted at 2026-09-28 10:49:58 on dist-test-slave-mn38"
server_key: "e998baefae0f5468961ca72fa4b1ea9a"
server_key_iv: "80f86d15ce65811bc714f7839a6dade8"
server_key_version: "encryptionkey@0"
I20260928 10:49:58.207392 31897 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 10:49:58.211448 31912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:58.212976 31897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 10:49:58.213163 31897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/wal
uuid: "50af76c70c884bb79ee941714462d551"
format_stamp: "Formatted at 2026-09-28 10:49:58 on dist-test-slave-mn38"
server_key: "e998baefae0f5468961ca72fa4b1ea9a"
server_key_iv: "80f86d15ce65811bc714f7839a6dade8"
server_key_version: "encryptionkey@0"
I20260928 10:49:58.213430 31897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:58.241398 31897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:58.246119 31897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:58.246415 31897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:58.266113 31897 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.254:44301
I20260928 10:49:58.266179 31963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.254:44301 every 8 connection(s)
I20260928 10:49:58.267577 31897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/data/info.pb
I20260928 10:49:58.271793 31964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:58.276072 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31897
I20260928 10:49:58.276273 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-0/wal/instance
I20260928 10:49:58.276800 9067 external_mini_cluster.cc:1428] Setting key c3b290c584257e42bc368d058e9bc0b0
I20260928 10:49:58.280712 31964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:49:58 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.253@KRBTEST.COM" created.
I20260928 10:49:58.306455 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.284901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.8.218.254:44301)
0928 10:49:58.286785 (+ 1884us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:58.286793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:58.286848 (+ 55us) server_negotiation.cc:407] Connection header received
0928 10:49:58.288702 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:58.288709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:58.289008 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:58.289252 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:58.290737 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.292035 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.293605 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.294245 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.294444 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:58.298587 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:58.298597 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:58.298611 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:58.298654 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:58.301762 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:58.302568 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:58.302575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:58.302578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:58.302769 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:58.303169 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:58.303173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:58.303175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:58.303773 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:58.303974 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:58.304547 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 10:49:58.305072 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":1648,"spinlock_wait_cycles":50944,"thread_start_us":1418,"threads_started":1}
I20260928 10:49:58.306499 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.284901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44301 (local address 127.0.0.1:44230)
0928 10:49:58.285877 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.285937 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:58.288081 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:58.289283 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:58.289291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:58.290121 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:58.290507 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:58.290530 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.292209 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.292215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:58.293393 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:58.293408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.294517 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.294525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:58.294720 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:58.295545 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:58.295578 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:58.298287 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:58.301933 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:58.301946 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:58.301961 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:58.302415 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:58.302889 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:58.302894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:58.302898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:58.303054 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:58.303910 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:58.303919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:58.304144 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 10:49:58.304385 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 10:49:58.304918 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":419,"threads_started":1}
I20260928 10:49:58.309798 31964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:58.311110 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.310574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50442)
0928 10:49:58.310791 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.310945 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 10:49:58.311476 31914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
I20260928 10:49:58.313509 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.312939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50454)
0928 10:49:58.313152 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.313272 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 10:49:58.313794 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:45801: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab.
Entry for principal kudu/127.8.218.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab.
I20260928 10:49:58.337273 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.253:45801
--webserver_interface=127.8.218.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab
--principal=kudu/127.8.218.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:44301,127.8.218.253:45801,127.8.218.252:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:49:58.363166 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } attempt: 1
I20260928 10:49:58.364434 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.363957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50468)
0928 10:49:58.364161 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.364288 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 10:49:58.365518 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.365121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50470)
0928 10:49:58.365261 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.365340 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 10:49:58.365710 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:45801: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
I20260928 10:49:58.433976 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } attempt: 2
I20260928 10:49:58.435266 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.434742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50478)
0928 10:49:58.435013 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.435108 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 10:49:58.436342 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.435938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50488)
0928 10:49:58.436085 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.436201 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 10:49:58.436542 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:45801: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
I20260928 10:49:58.527838 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } attempt: 3
I20260928 10:49:58.529066 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.528564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50492)
0928 10:49:58.528804 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.528904 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 10:49:58.530189 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.529805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50508)
0928 10:49:58.529965 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.530040 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 10:49:58.530368 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:45801: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
I20260928 10:49:58.673638 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } attempt: 4
I20260928 10:49:58.674865 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.674321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50514)
0928 10:49:58.674632 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.674718 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260928 10:49:58.675837 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.675471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50520)
0928 10:49:58.675613 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.675696 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 10:49:58.676021 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.253:45801: Network error: Client connection negotiation failed: client connection to 127.8.218.253:45801: connect: Connection refused (error 111)
W20260928 10:49:58.685825 31971 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:49:58.686187 31971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:58.686285 31971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:58.686406 31971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:58.694303 31971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:58.694418 31971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:58.694500 31971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:58.694568 31971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:58.694657 31971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:58.705312 31971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:44301,127.8.218.253:45801,127.8.218.252:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.253.keytab
--principal=kudu/127.8.218.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.253:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.218.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:58.706772 31971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:58.708212 31971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:58.720832 31976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:58.720741 31977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:58.720777 31979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:58.722199 31971 server_base.cc:1034] running on GCE node
I20260928 10:49:58.723029 31971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:58.724682 31971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:58.725960 31971 hybrid_clock.cc:648] HybridClock initialized: now 1790592598725871 us; error 61 us; skew 500 ppm
I20260928 10:49:58.726418 31971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:58 dist-test-slave-mn38 krb5kdc[31884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:58.729852 31971 init.cc:377] Logged in from keytab as kudu/127.8.218.253@KRBTEST.COM (short username kudu)
I20260928 10:49:58.731498 31971 webserver.cc:460] Webserver started at http://127.8.218.253:34373/ using document root <none> and password file <none>
I20260928 10:49:58.732280 31971 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:58.732396 31971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:58.732728 31971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:58.736017 31971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/data/instance:
uuid: "fb5e64071c5b4f108f27ae045715a999"
format_stamp: "Formatted at 2026-09-28 10:49:58 on dist-test-slave-mn38"
server_key: "bdf877e3aa1f234fae0965c8c48477d9"
server_key_iv: "56839320d180ca4543ed6222ad0185f8"
server_key_version: "encryptionkey@0"
I20260928 10:49:58.736887 31971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/wal/instance:
uuid: "fb5e64071c5b4f108f27ae045715a999"
format_stamp: "Formatted at 2026-09-28 10:49:58 on dist-test-slave-mn38"
server_key: "bdf877e3aa1f234fae0965c8c48477d9"
server_key_iv: "56839320d180ca4543ed6222ad0185f8"
server_key_version: "encryptionkey@0"
I20260928 10:49:58.742592 31971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 10:49:58.746353 31986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:58.747912 31971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 10:49:58.748106 31971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/wal
uuid: "fb5e64071c5b4f108f27ae045715a999"
format_stamp: "Formatted at 2026-09-28 10:49:58 on dist-test-slave-mn38"
server_key: "bdf877e3aa1f234fae0965c8c48477d9"
server_key_iv: "56839320d180ca4543ed6222ad0185f8"
server_key_version: "encryptionkey@0"
I20260928 10:49:58.748346 31971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:58.803908 31971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:58.808677 31971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:58.808945 31971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:58.827610 32037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.253:45801 every 8 connection(s)
I20260928 10:49:58.827605 31971 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.253:45801
I20260928 10:49:58.829289 31971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/data/info.pb
I20260928 10:49:58.833246 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:58.836333 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 31971
I20260928 10:49:58.836494 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-1/wal/instance
I20260928 10:49:58.836943 9067 external_mini_cluster.cc:1428] Setting key 97d25dc98035096584234fe2eeae5df3
I20260928 10:49:58.840303 32038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:49:58 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.218.252@KRBTEST.COM; defaulting to no policy
I20260928 10:49:58.863214 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.843173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.8.218.254:44301)
0928 10:49:58.843688 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:58.843694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:58.844776 (+ 1082us) server_negotiation.cc:407] Connection header received
0928 10:49:58.847291 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:58.847295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:58.847371 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:58.847473 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:58.849549 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.850268 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.852349 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.852807 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.852906 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:58.857087 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:58.857095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:58.857100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:58.857161 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:58.859834 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:58.860845 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:58.860849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:58.860851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:58.860978 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:58.861490 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:58.861495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:58.861497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:58.861836 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:58.861953 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:58.862707 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 10:49:58.862939 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":250,"threads_started":1}
I20260928 10:49:58.864125 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.843763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44301 (local address 127.0.0.1:44236)
0928 10:49:58.844569 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.844622 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:58.846954 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:58.847760 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:58.847772 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:58.848466 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:58.849277 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:58.849301 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.850421 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.850426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:58.852213 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:58.852227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.853032 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.853038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:58.853311 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:58.854065 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:58.854093 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:58.856762 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:58.860017 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:58.860028 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:58.860048 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:58.860686 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:58.861083 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:58.861087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:58.861091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:58.861339 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:58.861965 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:58.861974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:58.862365 (+ 391us) client_negotiation.cc:769] Sending connection context
0928 10:49:58.862603 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 10:49:58.863036 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":327,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.218.252@KRBTEST.COM" created.
I20260928 10:49:58.866037 32038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:49:58 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
I20260928 10:49:58.880656 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.866889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50524)
0928 10:49:58.867053 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.867071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:58.867173 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:58.867942 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:58.867946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:58.868377 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:58.868505 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:58.868513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.869900 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.869905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:58.870748 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:58.870758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.871546 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.871551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:58.871709 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:58.872340 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:58.872357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:58.874581 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:58.878185 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:58.878189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:58.878192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:58.878529 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:58.878981 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:58.878985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:58.878988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:58.879139 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:58.879832 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:58.879836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:58.879957 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 10:49:58.880140 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 10:49:58.880380 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 10:49:58.881126 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.866956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.8.218.253:45801)
0928 10:49:58.867521 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:58.867527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:58.867549 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:49:58.867593 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:58.867599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:58.867803 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:58.867940 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:58.868655 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.869780 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.870875 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.871417 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.871522 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:58.874819 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:58.874834 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:58.874852 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:58.874887 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:58.878052 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:58.878664 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:58.878673 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:58.878676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:58.878820 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:58.879242 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:58.879246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:58.879249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:58.879670 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:58.879885 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:58.880310 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 10:49:58.880573 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":277,"threads_started":1}
I20260928 10:49:58.883685 32038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:58.884775 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.884301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39902)
0928 10:49:58.884493 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.884622 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 10:49:58.885004 31988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
I20260928 10:49:58.886529 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.886209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39910)
0928 10:49:58.886347 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.886425 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 10:49:58.886714 32038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.218.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab.
Entry for principal kudu/127.8.218.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab.
I20260928 10:49:58.897631 9067 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3zclSv/build/asan/bin/kudu
/tmp/dist-test-task3zclSv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.252:38469
--webserver_interface=127.8.218.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab
--principal=kudu/127.8.218.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.218.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.218.254:44301,127.8.218.253:45801,127.8.218.252:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 10:49:58.936012 32038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 1
I20260928 10:49:58.937528 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.936845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39914)
0928 10:49:58.937162 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.937298 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 10:49:58.938776 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.938364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39924)
0928 10:49:58.938530 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.938628 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 10:49:58.939028 32038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
I20260928 10:49:58.975281 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } attempt: 5
Sep 28 10:49:58 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
I20260928 10:49:58.989984 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.976217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.8.218.253:45801)
0928 10:49:58.976393 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:58.976397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:58.976416 (+ 19us) server_negotiation.cc:407] Connection header received
0928 10:49:58.976467 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:58.976471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:58.976541 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:58.976625 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:58.977629 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.978318 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.979985 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.980447 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.980552 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:58.984273 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:58.984279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:58.984283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:58.984320 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:58.987087 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:58.987949 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:58.987953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:58.987955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:58.988090 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:58.988518 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:58.988523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:58.988525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:58.988870 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:58.988994 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:58.989527 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 10:49:58.989740 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 10:49:58.990126 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.975940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50526)
0928 10:49:58.976163 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.976185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:58.976337 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:58.976699 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:58.976705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:58.977189 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:58.977425 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:58.977478 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.978460 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.978466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:58.979820 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:58.979839 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:58.980625 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:58.980631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:58.980870 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:58.981551 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:58.981573 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:58.984043 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:58.987280 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:58.987287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:58.987291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:58.987814 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:58.988197 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:58.988210 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:58.988214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:58.988382 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:58.988996 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:58.989000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:58.989208 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 10:49:58.989371 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 10:49:58.989772 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 10:49:58.991308 31964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:58.992430 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.991899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39934)
0928 10:49:58.992110 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.992238 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 10:49:58.993460 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:58.993039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39946)
0928 10:49:58.993193 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:58.993311 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 10:49:58.993683 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
I20260928 10:49:59.007282 32038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 2
I20260928 10:49:59.008489 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.007957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39948)
0928 10:49:59.008168 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.008292 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 10:49:59.009584 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.009157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39956)
0928 10:49:59.009320 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.009407 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 10:49:59.009789 32038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
I20260928 10:49:59.044957 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 1
I20260928 10:49:59.046341 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.045770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39966)
0928 10:49:59.045996 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.046116 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 10:49:59.047392 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.046988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39972)
0928 10:49:59.047150 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.047248 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 10:49:59.047648 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
I20260928 10:49:59.101035 32038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 3
I20260928 10:49:59.102329 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.101756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:39984)
0928 10:49:59.102022 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.102134 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 10:49:59.103427 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.102954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40000)
0928 10:49:59.103186 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.103279 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260928 10:49:59.103645 32038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
I20260928 10:49:59.115859 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 2
I20260928 10:49:59.117118 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.116476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40012)
0928 10:49:59.116749 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.116871 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 10:49:59.118263 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.117869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40018)
0928 10:49:59.118031 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.118124 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 10:49:59.118522 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
I20260928 10:49:59.224764 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 3
I20260928 10:49:59.226117 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.225543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40022)
0928 10:49:59.225785 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.225896 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 10:49:59.227394 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.226943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40024)
0928 10:49:59.227146 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.227243 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 10:49:59.227656 31964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
I20260928 10:49:59.246902 32038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 4
I20260928 10:49:59.248154 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.247581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40026)
0928 10:49:59.247821 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.247936 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 10:49:59.249392 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.248972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40034)
0928 10:49:59.249147 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.249241 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 10:49:59.249346 32046 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 10:49:59.249655 32038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.218.252:38469: Network error: Client connection negotiation failed: client connection to 127.8.218.252:38469: connect: Connection refused (error 111)
W20260928 10:49:59.249843 32046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 10:49:59.249924 32046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 10:49:59.250007 32046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 10:49:59.258018 32046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 10:49:59.258141 32046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 10:49:59.258203 32046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 10:49:59.258252 32046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 10:49:59.258312 32046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 10:49:59.268738 32046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.218.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.218.254:44301,127.8.218.253:45801,127.8.218.252:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/krb5kdc/kudu_127.8.218.252.keytab
--principal=kudu/127.8.218.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.218.252:38469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.218.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 09:56:20 UTC on ccycloud.quasar-pbuesr.root.comops.site
build id 4676955
ASAN enabled
I20260928 10:49:59.270120 32046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 10:49:59.271405 32046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 10:49:59.283380 32052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:59.283345 32054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 10:49:59.283420 32051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 10:49:59.284046 32046 server_base.cc:1034] running on GCE node
I20260928 10:49:59.284920 32046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 10:49:59.286460 32046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 10:49:59.287698 32046 hybrid_clock.cc:648] HybridClock initialized: now 1790592599287616 us; error 55 us; skew 500 ppm
I20260928 10:49:59.288028 32046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 10:49:59.291343 32046 init.cc:377] Logged in from keytab as kudu/127.8.218.252@KRBTEST.COM (short username kudu)
I20260928 10:49:59.292886 32046 webserver.cc:460] Webserver started at http://127.8.218.252:46581/ using document root <none> and password file <none>
I20260928 10:49:59.293674 32046 fs_manager.cc:359] Metadata directory not provided
I20260928 10:49:59.293776 32046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 10:49:59.294060 32046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 10:49:59.297219 32046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/data/instance:
uuid: "5beccdff3db34c0b92af82f03023bcd1"
format_stamp: "Formatted at 2026-09-28 10:49:59 on dist-test-slave-mn38"
server_key: "5e9e8bc1f7c1ce7afc71f62873b5556b"
server_key_iv: "7fc68ed5212e5569796e9ae337689adc"
server_key_version: "encryptionkey@0"
I20260928 10:49:59.298105 32046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/wal/instance:
uuid: "5beccdff3db34c0b92af82f03023bcd1"
format_stamp: "Formatted at 2026-09-28 10:49:59 on dist-test-slave-mn38"
server_key: "5e9e8bc1f7c1ce7afc71f62873b5556b"
server_key_iv: "7fc68ed5212e5569796e9ae337689adc"
server_key_version: "encryptionkey@0"
I20260928 10:49:59.303920 32046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 10:49:59.307916 32061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 10:49:59.309644 32046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 10:49:59.309883 32046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/data,/tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/wal
uuid: "5beccdff3db34c0b92af82f03023bcd1"
format_stamp: "Formatted at 2026-09-28 10:49:59 on dist-test-slave-mn38"
server_key: "5e9e8bc1f7c1ce7afc71f62873b5556b"
server_key_iv: "7fc68ed5212e5569796e9ae337689adc"
server_key_version: "encryptionkey@0"
I20260928 10:49:59.310125 32046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 10:49:59.366139 32046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 10:49:59.370961 32046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 10:49:59.371243 32046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 10:49:59.389959 32046 rpc_server.cc:307] RPC server started. Bound to: 127.8.218.252:38469
I20260928 10:49:59.389976 32112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.218.252:38469 every 8 connection(s)
I20260928 10:49:59.391419 32046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/data/info.pb
I20260928 10:49:59.394703 9067 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3zclSv/build/asan/bin/kudu as pid 32046
I20260928 10:49:59.394871 9067 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3zclSv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790592230404718-9067-0/minicluster-data/master-2/wal/instance
I20260928 10:49:59.395306 9067 external_mini_cluster.cc:1428] Setting key 74b4a1ebddebe450d65bdc02599f7f41
I20260928 10:49:59.395725 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 10:49:59.404907 31964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 4
I20260928 10:49:59.406065 32113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.254@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
I20260928 10:49:59.424455 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.405686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40050)
0928 10:49:59.405943 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.405966 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:59.406104 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:59.407332 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:59.407340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:59.407866 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:59.408037 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.408067 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.409886 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.409893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.411500 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.411528 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.412886 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.412895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.413232 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.414071 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:59.414100 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:59.416971 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:59.421283 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.421290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.421294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.421915 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.422271 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.422276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.422280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.422453 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.423623 (+ 1170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:59.423629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:59.423814 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 10:49:59.423936 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 10:49:59.424183 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
I20260928 10:49:59.426517 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.397131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.8.218.252:38469)
0928 10:49:59.397980 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.397992 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.398036 (+ 44us) server_negotiation.cc:407] Connection header received
0928 10:49:59.400515 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.400544 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.401074 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.401524 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.403481 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.404951 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.406868 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.408133 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.408379 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.414516 (+ 6137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.414530 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.414561 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.416466 (+ 1905us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x60c0000347b8. stack: 00007f6938a2766e 00007f6938a27430 00007f6937aae5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f6938a1d0fd 00007f6938a1cd1e 00007f693f534664 00007f693f536b75 00007f693f55070e 00007f693f54a01e 00007f693f5438a5 00007f693f4b1ea5 00007f693f4b0eff 00007f693f4e482f
0928 10:49:59.416513 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.421013 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.421902 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.421909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.421927 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.422164 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.422568 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.422582 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.422585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.423483 (+ 898us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.424727 (+ 1244us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.424878 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.425089 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":627,"spinlock_wait_cycles":2411392,"thread_start_us":381,"threads_started":1}
I20260928 10:49:59.426539 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.406150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.8.218.252:38469)
0928 10:49:59.407000 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.407005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.407027 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:49:59.407071 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.407075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.407142 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.407237 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.408276 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.409665 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.411786 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.412532 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.413736 (+ 1204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.417215 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.417220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.417224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.417259 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.421080 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.422035 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.422040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.422042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.422156 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.422574 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.422578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.422581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.423483 (+ 902us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.423688 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.424224 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.424904 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":703,"spinlock_wait_cycles":10624,"thread_start_us":490,"threads_started":1}
I20260928 10:49:59.434199 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.409909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.254:44301 (local address 127.0.0.1:44246)
0928 10:49:59.415871 (+ 5962us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.415928 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:59.416107 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:59.416765 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:59.416775 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:59.417919 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:59.418404 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.418419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.420036 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.420042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.422022 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.422038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.422945 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.422953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.423141 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.424900 (+ 1759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:59.424917 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:59.427314 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:59.431247 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.431261 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.431268 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.431703 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.432106 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.432111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.432115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.432257 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.432866 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:59.432875 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:59.433160 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 10:49:59.433344 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 10:49:59.433760 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5683,"thread_start_us":5537,"threads_started":1}
I20260928 10:49:59.434211 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.409820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.8.218.254:44301)
0928 10:49:59.415554 (+ 5734us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.415559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.416252 (+ 693us) server_negotiation.cc:407] Connection header received
0928 10:49:59.416326 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.416332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.416415 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.416526 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.418624 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.419822 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.422153 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.422750 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.422858 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.427637 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.427644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.427648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.427686 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.431085 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.431848 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.431852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.431854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.431984 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.432376 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.432380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.432382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.432733 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.432871 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.433514 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.433786 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5555,"thread_start_us":5335,"threads_started":1}
I20260928 10:49:59.436625 31964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551: Bootstrap starting.
I20260928 10:49:59.438177 32113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:59.441742 31964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:59.443280 31964 log.cc:824] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551: Log is configured to *not* fsync() on all Append() calls
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.254@KRBTEST.COM
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
I20260928 10:49:59.447310 31964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551: No bootstrap required, opened a new log
I20260928 10:49:59.452234 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.430177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.8.218.254:44301)
0928 10:49:59.434507 (+ 4330us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.434511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.434524 (+ 13us) server_negotiation.cc:407] Connection header received
0928 10:49:59.434577 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.434582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.434658 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.434750 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.435995 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.436788 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.438427 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.439192 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.439322 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.444924 (+ 5602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.444932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.444936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.444984 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.448637 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.449751 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.449757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.449761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.449928 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.450372 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.450377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.450380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.450855 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.451161 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.451569 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.451854 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4244,"thread_start_us":8423,"threads_started":1}
I20260928 10:49:59.454197 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.439076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.8.218.253:45801)
0928 10:49:59.439308 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.439314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.439461 (+ 147us) server_negotiation.cc:407] Connection header received
0928 10:49:59.439627 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.439633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.439709 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.439804 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.440722 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.441418 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.442725 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.443308 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.443459 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.447155 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.447163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.447167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.447214 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.450496 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.451378 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.451385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.451389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.451564 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.452297 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.452302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.452305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.452689 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.452826 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.453533 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.453806 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 10:49:59.454481 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.439089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.253:45801 (local address 127.0.0.1:50530)
0928 10:49:59.439371 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.439390 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:59.439511 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:59.439842 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:59.439847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:59.440369 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:59.440560 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.440572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.441601 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.441607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.442564 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.442579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.443518 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.443525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.443727 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.444387 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:59.444411 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:59.446896 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:59.450692 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.450698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.450703 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.451249 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.451716 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.451721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.451725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.451909 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.452939 (+ 1030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:59.452944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:59.453136 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 10:49:59.453301 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 10:49:59.454134 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260928 10:49:59.456012 32113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } has no permanent_uuid. Determining permanent_uuid...
I20260928 10:49:59.458345 31964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } }
I20260928 10:49:59.458874 31964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50af76c70c884bb79ee941714462d551, State: Initialized, Role: FOLLOWER
I20260928 10:49:59.459573 31964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } }
I20260928 10:49:59.460644 32123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.461194 32123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:59.462172 31964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.252@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.218.253@KRBTEST.COM
I20260928 10:49:59.473013 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.454627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.8.218.253:45801)
0928 10:49:59.455236 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.455241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.455263 (+ 22us) server_negotiation.cc:407] Connection header received
0928 10:49:59.455320 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.455325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.455401 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.455513 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.458375 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.459646 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.461499 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.462158 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.462305 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.467424 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.467430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.467433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.467472 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.470280 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.470988 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.470992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.470995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.471129 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.471518 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.471523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.471524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.471813 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.472111 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.472388 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.472623 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":276,"threads_started":1}
I20260928 10:49:59.473268 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.456946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.8.218.252:38469)
0928 10:49:59.457192 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.457196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.457371 (+ 175us) server_negotiation.cc:407] Connection header received
0928 10:49:59.457577 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.457582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.457647 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.457724 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.458796 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.459593 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.461161 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.461760 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.461843 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.466088 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.466097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.466101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.466141 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.470143 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.471009 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.471014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.471018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.471155 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.471537 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.471542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.471546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.471979 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.472127 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.472640 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.472910 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 10:49:59.474113 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.456908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40064)
0928 10:49:59.457280 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.457294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:59.457415 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:59.457760 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:59.457765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:59.458420 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:59.458635 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.458645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.459756 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.459760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.461012 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.461029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.462038 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.462042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.462169 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.462921 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:59.462943 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:59.465850 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:59.470332 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.470338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.470343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.470883 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.471295 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.471300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.471304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.471439 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.472213 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:59.472217 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:59.472363 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 10:49:59.473467 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0928 10:49:59.473779 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304}
I20260928 10:49:59.478458 9067 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 10:49:59.479840 32113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1: Bootstrap starting.
I20260928 10:49:59.481760 31939 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "50af76c70c884bb79ee941714462d551"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44262
I20260928 10:49:59.482206 31939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 10:49:59.482407 31939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:59.483736 32113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:59.484818 32113 log.cc:824] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1: Log is configured to *not* fsync() on all Append() calls
W20260928 10:49:59.488621 32134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 10:49:59.488811 32134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 10:49:59.490604 31939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } }
I20260928 10:49:59.492483 31939 leader_election.cc:288] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [CANDIDATE]: Term 1 election: Requested vote from peers fb5e64071c5b4f108f27ae045715a999 (127.8.218.253:45801), 5beccdff3db34c0b92af82f03023bcd1 (127.8.218.252:38469)
I20260928 10:49:59.494470 32113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1: No bootstrap required, opened a new log
I20260928 10:49:59.485409 31897 master_runner.cc:186] Error getting master registration for 127.8.218.253:45801: OK, instance_id { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" instance_seqno: 1790592598822282 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 10:49:59.496596 31897 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 10:49:59.499650 31918 tablet.cc:2369] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:59.506973 32113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } }
I20260928 10:49:59.507503 32113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beccdff3db34c0b92af82f03023bcd1, State: Initialized, Role: FOLLOWER
I20260928 10:49:59.508087 32113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } }
I20260928 10:49:59.515866 32143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.516474 32143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:59.518738 32113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:59.518844 32088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50af76c70c884bb79ee941714462d551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5beccdff3db34c0b92af82f03023bcd1"
I20260928 10:49:59.519457 32088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:59.523541 32088 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50af76c70c884bb79ee941714462d551 in term 1.
I20260928 10:49:59.524437 31914 leader_election.cc:302] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50af76c70c884bb79ee941714462d551, 5beccdff3db34c0b92af82f03023bcd1; no voters:
I20260928 10:49:59.525074 32123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 10:49:59.540043 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.499152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.8.218.253:45801)
0928 10:49:59.499370 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.499377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.507654 (+ 8277us) server_negotiation.cc:407] Connection header received
0928 10:49:59.508867 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.508875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.508974 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.509081 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.516487 (+ 7406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.517681 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.521728 (+ 4047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.522446 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.522562 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.532682 (+ 10120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.532692 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.532697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.532758 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.536511 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.537458 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.537466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.537470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.537677 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.538181 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.538186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.538189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.538574 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.538831 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.539431 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.539695 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 10:49:59.545665 32123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [term 1 LEADER]: Becoming Leader. State: Replica: 50af76c70c884bb79ee941714462d551, State: Running, Role: LEADER
I20260928 10:49:59.546834 32123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } }
I20260928 10:49:59.548825 32038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } attempt: 5
I20260928 10:49:59.552508 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.499155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.8.218.254:44301)
0928 10:49:59.499353 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.499360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.512723 (+ 13363us) server_negotiation.cc:407] Connection header received
0928 10:49:59.513256 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.513264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.513375 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.513499 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.519930 (+ 6431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.521471 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.527594 (+ 6123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.528471 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.528617 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.540040 (+ 11423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.540048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.540053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.540103 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.545056 (+ 4953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.546371 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.546377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.546381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.546551 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.547351 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.547357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.547361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.548126 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.548721 (+ 595us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.551383 (+ 2662us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.551829 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 10:49:59.553661 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.508560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.8.218.252:38469)
0928 10:49:59.508761 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.508767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.530094 (+ 21327us) server_negotiation.cc:407] Connection header received
0928 10:49:59.530420 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.530426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.530520 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.530604 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.534651 (+ 4047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.535795 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.541696 (+ 5901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.542344 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.542452 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.545612 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.545621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.545626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.545673 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.549571 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.550412 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.550419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.550422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.550601 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.551041 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.551059 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.551062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.551442 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.551734 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.552509 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.553140 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 10:49:59.556789 32146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50af76c70c884bb79ee941714462d551. Latest consensus state: current_term: 1 leader_uuid: "50af76c70c884bb79ee941714462d551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.557298 32146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: This master's current role is: LEADER
Sep 28 10:49:59 dist-test-slave-mn38 krb5kdc[31884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790592598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.218.253@KRBTEST.COM for kudu/127.8.218.252@KRBTEST.COM
I20260928 10:49:59.565042 32151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 10:49:59.567612 32151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 10:49:59.571117 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.549517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.218.252:38469 (local address 127.0.0.1:40088)
0928 10:49:59.549917 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 10:49:59.549938 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 10:49:59.550044 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 10:49:59.550960 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 10:49:59.550966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 10:49:59.551483 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 10:49:59.551716 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.551730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.553257 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.553264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.554807 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 10:49:59.554828 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.557143 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.557151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 10:49:59.557391 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.558447 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 10:49:59.558472 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 10:49:59.561525 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 10:49:59.567836 (+ 6311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.567842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.567846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.568475 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.568989 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 10:49:59.568995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 10:49:59.568998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 10:49:59.569209 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 10:49:59.570010 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 10:49:59.570016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 10:49:59.570204 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 10:49:59.570358 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 10:49:59.570704 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259}
I20260928 10:49:59.572368 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:49:59.550269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.8.218.252:38469)
0928 10:49:59.550464 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.550471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.550492 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:49:59.550549 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.550553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.550634 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.550743 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.551902 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.553052 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.555012 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:49:59.556742 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:49:59.556903 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:49:59.561796 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 10:49:59.561804 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 10:49:59.561809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 10:49:59.561854 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 10:49:59.567603 (+ 5749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.568660 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.568665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.568669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.568844 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 10:49:59.569353 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 10:49:59.569358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 10:49:59.569361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 10:49:59.569785 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 10:49:59.569939 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 10:49:59.571624 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0928 10:49:59.571961 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 10:49:59.580013 32038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999: Bootstrap starting.
I20260928 10:49:59.582080 32088 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [term 1 FOLLOWER]: Refusing update from remote peer 50af76c70c884bb79ee941714462d551: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:59.583547 32146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:59.586206 32038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999: Neither blocks nor log segments found. Creating new log.
I20260928 10:49:59.587603 32038 log.cc:824] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999: Log is configured to *not* fsync() on all Append() calls
I20260928 10:49:59.596122 32038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999: No bootstrap required, opened a new log
I20260928 10:49:59.610492 32143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50af76c70c884bb79ee941714462d551. Latest consensus state: current_term: 1 leader_uuid: "50af76c70c884bb79ee941714462d551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.610903 32143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:59.614144 32038 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } }
I20260928 10:49:59.614717 32038 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5e64071c5b4f108f27ae045715a999, State: Initialized, Role: FOLLOWER
I20260928 10:49:59.615351 32038 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } }
I20260928 10:49:59.622762 32152 mvcc.cc:205] Tried to move back new op lower bound from 7334267287863914496 to 7334267287750135808. Current Snapshot: MvccSnapshot[applied={T|T < 7334267287863914496}]
W20260928 10:49:59.623588 32067 tablet.cc:2369] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 10:49:59.625017 32158 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.625785 32158 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:59.626257 32143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50af76c70c884bb79ee941714462d551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.626554 32143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:59.628588 32038 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 10:49:59.628639 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50af76c70c884bb79ee941714462d551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fb5e64071c5b4f108f27ae045715a999"
I20260928 10:49:59.629129 32012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 10:49:59.635526 32146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50af76c70c884bb79ee941714462d551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.636067 32012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [term 1 FOLLOWER]: Refusing update from remote peer 50af76c70c884bb79ee941714462d551: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 10:49:59.636337 32146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:59.640673 32013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50af76c70c884bb79ee941714462d551 in term 1.
I20260928 10:49:59.643288 32146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 10:49:59.659425 32151 catalog_manager.cc:1347] Generated new cluster ID: 59c54c3aeb504f05a55288353b917054
I20260928 10:49:59.659575 32151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 10:49:59.700213 32158 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50af76c70c884bb79ee941714462d551. Latest consensus state: current_term: 1 leader_uuid: "50af76c70c884bb79ee941714462d551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.700557 32158 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:59.705329 32146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50af76c70c884bb79ee941714462d551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.705727 32146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551 [sys.catalog]: This master's current role is: LEADER
I20260928 10:49:59.708824 32175 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50af76c70c884bb79ee941714462d551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50af76c70c884bb79ee941714462d551" member_type: VOTER last_known_addr { host: "127.8.218.254" port: 44301 } } peers { permanent_uuid: "fb5e64071c5b4f108f27ae045715a999" member_type: VOTER last_known_addr { host: "127.8.218.253" port: 45801 } } peers { permanent_uuid: "5beccdff3db34c0b92af82f03023bcd1" member_type: VOTER last_known_addr { host: "127.8.218.252" port: 38469 } } }
I20260928 10:49:59.709296 32175 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 10:49:59.710796 32180 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:49:59.710845 32179 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 10:49:59.721050 32151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 10:49:59.724499 32151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 10:49:59.738034 32151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50af76c70c884bb79ee941714462d551: Generated new TSK 0
I20260928 10:49:59.739428 32151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 10:49:59.811834 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.808312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.8.218.253:45801)
0928 10:49:59.808513 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.808520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.809166 (+ 646us) server_negotiation.cc:407] Connection header received
0928 10:49:59.809555 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.809562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.809666 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.809775 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.811469 (+ 1694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50558: BlockingRecv error: recv got EOF from 127.0.0.1:50558 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 10:49:59.815065 32118 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.808175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.8.218.254:44301)
0928 10:49:59.808467 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.808475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.812755 (+ 4280us) server_negotiation.cc:407] Connection header received
0928 10:49:59.813083 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.813091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.813199 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.813288 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.814678 (+ 1390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44278: BlockingRecv error: recv got EOF from 127.0.0.1:44278 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 10:49:59.818372 32116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.809605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.8.218.252:38469)
0928 10:49:59.809794 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.809801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.815967 (+ 6166us) server_negotiation.cc:407] Connection header received
0928 10:49:59.816297 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.816304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.816414 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.816503 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.818043 (+ 1540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40102: BlockingRecv error: recv got EOF from 127.0.0.1:40102 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 10:49:59.827044 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.824794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.8.218.253:45801)
0928 10:49:59.825013 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.825021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.825492 (+ 471us) server_negotiation.cc:407] Connection header received
0928 10:49:59.825771 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.825775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.825880 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.825970 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.826798 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50574: BlockingRecv error: recv got EOF from 127.0.0.1:50574 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 10:49:59.839782 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.835754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.8.218.253:45801)
0928 10:49:59.835955 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.835961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.838177 (+ 2216us) server_negotiation.cc:407] Connection header received
0928 10:49:59.838377 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.838383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.838484 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.838608 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.839501 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50580: BlockingRecv error: recv got EOF from 127.0.0.1:50580 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 10:49:59.854195 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.851634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.8.218.253:45801)
0928 10:49:59.851828 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.851835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.852407 (+ 572us) server_negotiation.cc:407] Connection header received
0928 10:49:59.852667 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.852673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.852785 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.852913 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.853913 (+ 1000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50584: BlockingRecv error: recv got EOF from 127.0.0.1:50584 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 10:49:59.864197 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.861754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.8.218.253:45801)
0928 10:49:59.861985 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.861991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.862441 (+ 450us) server_negotiation.cc:407] Connection header received
0928 10:49:59.862692 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.862698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.862783 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.862904 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.863902 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50588: BlockingRecv error: recv got EOF from 127.0.0.1:50588 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 10:49:59.878341 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.876363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.8.218.253:45801)
0928 10:49:59.876565 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.876571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.876846 (+ 275us) server_negotiation.cc:407] Connection header received
0928 10:49:59.877018 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.877023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.877136 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.877255 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.878059 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50590: BlockingRecv error: recv got EOF from 127.0.0.1:50590 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 10:49:59.900156 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.898085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.8.218.253:45801)
0928 10:49:59.898311 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.898318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.898609 (+ 291us) server_negotiation.cc:407] Connection header received
0928 10:49:59.898790 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.898795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.898898 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.899029 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.899859 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50594: BlockingRecv error: recv got EOF from 127.0.0.1:50594 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 10:49:59.937595 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:49:59.935511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.8.218.253:45801)
0928 10:49:59.935705 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 10:49:59.935713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:49:59.936164 (+ 451us) server_negotiation.cc:407] Connection header received
0928 10:49:59.936371 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:49:59.936376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:49:59.936455 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:49:59.936544 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:49:59.937345 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50598: BlockingRecv error: recv got EOF from 127.0.0.1:50598 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 10:50:00.008427 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:50:00.006248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.8.218.253:45801)
0928 10:50:00.006476 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:00.006482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:00.006844 (+ 362us) server_negotiation.cc:407] Connection header received
0928 10:50:00.007040 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:00.007044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:00.007135 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:00.007225 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:00.008130 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50606: BlockingRecv error: recv got EOF from 127.0.0.1:50606 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 10:50:00.142067 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:50:00.140206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.8.218.253:45801)
0928 10:50:00.140404 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:00.140408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:00.140705 (+ 297us) server_negotiation.cc:407] Connection header received
0928 10:50:00.140867 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:00.140872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:00.140951 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:00.141052 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:00.141887 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50620: BlockingRecv error: recv got EOF from 127.0.0.1:50620 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 10:50:00.405787 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 10:50:00.403790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.8.218.253:45801)
0928 10:50:00.403978 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:00.403983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:00.404239 (+ 256us) server_negotiation.cc:407] Connection header received
0928 10:50:00.404483 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:00.404487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:00.404564 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:00.404651 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 10:50:00.405554 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50634: BlockingRecv error: recv got EOF from 127.0.0.1:50634 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 10:50:00.716326 32180 catalog_manager.cc:1259] Loaded cluster ID: 59c54c3aeb504f05a55288353b917054
I20260928 10:50:00.716459 32180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1: loading cluster ID for follower catalog manager: success
I20260928 10:50:00.716622 32179 catalog_manager.cc:1259] Loaded cluster ID: 59c54c3aeb504f05a55288353b917054
I20260928 10:50:00.716900 32179 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999: loading cluster ID for follower catalog manager: success
I20260928 10:50:00.721181 32180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1: acquiring CA information for follower catalog manager: success
I20260928 10:50:00.723340 32179 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999: acquiring CA information for follower catalog manager: success
I20260928 10:50:00.724175 32180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5beccdff3db34c0b92af82f03023bcd1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:50:00.726393 32179 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb5e64071c5b4f108f27ae045715a999: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 10:50:00.949172 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:00.927266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.8.218.253:45801)
0928 10:50:00.939671 (+ 12405us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:00.939679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:00.940451 (+ 772us) server_negotiation.cc:407] Connection header received
0928 10:50:00.940689 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:00.940696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:00.940921 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:00.941089 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:00.941750 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.943050 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.945274 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.946205 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.946412 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:00.947221 (+ 809us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:00.948115 (+ 894us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:00.948304 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:00.948481 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 10:50:00.948716 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12199,"thread_start_us":12431,"threads_started":1}
I20260928 10:50:00.970922 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:00.957260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40116 (local address 127.8.218.252:38469)
0928 10:50:00.962100 (+ 4840us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:00.962108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:00.962138 (+ 30us) server_negotiation.cc:407] Connection header received
0928 10:50:00.962369 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:00.962378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:00.962716 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:00.963184 (+ 468us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:00.963906 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.965186 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.967428 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.968214 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.968379 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:00.969095 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:00.970024 (+ 929us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:00.970192 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:00.970352 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 10:50:00.970452 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4637,"thread_start_us":4368,"threads_started":1}
I20260928 10:50:00.989734 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:00.980102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.8.218.253:45801)
0928 10:50:00.980300 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:00.980306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:00.981642 (+ 1336us) server_negotiation.cc:407] Connection header received
0928 10:50:00.981930 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:00.981936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:00.982035 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:00.982107 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:00.982901 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.984076 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.986689 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.987381 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.987504 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:00.988238 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:00.988433 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:00.988512 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:00.989221 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 10:50:00.989350 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 10:50:00.997493 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:00.980217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.8.218.254:44301)
0928 10:50:00.989847 (+ 9630us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:00.989853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:00.989874 (+ 21us) server_negotiation.cc:407] Connection header received
0928 10:50:00.989993 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:00.989999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:00.990251 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:00.990446 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:00.991096 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.992124 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.994311 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.994977 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.995660 (+ 683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:00.995866 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:00.996353 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:00.996976 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:00.997034 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 10:50:00.997131 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9481,"thread_start_us":9362,"threads_started":1}
I20260928 10:50:00.997613 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 10:50:00.981567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.8.218.252:38469)
0928 10:50:00.989868 (+ 8301us) server_negotiation.cc:206] Beginning negotiation
0928 10:50:00.989875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 10:50:00.989898 (+ 23us) server_negotiation.cc:407] Connection header received
0928 10:50:00.989973 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 10:50:00.989978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 10:50:00.990075 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 10:50:00.990929 (+ 854us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 10:50:00.991085 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.992400 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.994663 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 10:50:00.995374 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 10:50:00.995556 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 10:50:00.996155 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 10:50:00.996304 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 10:50:00.996390 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 10:50:00.996717 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 10:50:00.997261 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8150}
I20260928 10:50:01.003697 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31897
I20260928 10:50:01.005014 31959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:01.168486 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31897
W20260928 10:50:01.185822 31987 connection.cc:535] server connection from 127.0.0.1:50526 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 10:50:01.185895 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 31971
I20260928 10:50:01.187100 32033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:01.296339 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 31971
I20260928 10:50:01.313687 9067 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3zclSv/build/asan/bin/kudu pid 32046
I20260928 10:50:01.314903 32108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 10:50:01.425127 9067 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3zclSv/build/asan/bin/kudu with pid 32046
2026-09-28T10:50:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4939 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9438 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (370977 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST